Case: Implementing Sync & Video Transcoding on an Educational Domain Project

We at SDH have already performed plenty of both complex and easy tasks. Sometimes projects have some similar, basic things. Features mentioned in the article are completely unlike others.

We at SDH have already performed plenty of both complex and easy tasks. Sometimes projects have some similar, basic things. Features mentioned in the article are completely unlike others.

• The project represents an educational platform with 50,000 users. 
• The region is the USA.

Description of the task

The customer set us the task to simplify and facilitate the uploading of educational information packages to the site. The package includes educational content for the teacher, which helps conduct classes with students, a methodical material.

The person responsible for creating the training content (let's call him Harry) has to upload it to the site. By that time, the customer already had a Drupal-based admin panel, that was not very convenient. Harry creates a package of HTML pages. These HTML pages have a bright design, full of animations and videos.

Our task was to make that process easier for Harry to easily upload materials for teachers without using FTP and Drupal.

Challenges

During the implementation of the project, we solved the following main problems:

• the inconvenience of working with the Drupal admin panel;
• voluminous content packages;
• issues regarding playback of animations and video files of various formats; 
• video transcoding to convert video files into streaming video.

Necessary functionality

It took us more than two months to write the entire project, including sections and functionality:

1. Users;
2. Environments;
3. Packages;
4. Logging of all events in the admin;
5. Access rights in the admin area;
6. Collection of analytics on the user side;
7. Synchronization of the database of the web application and the Dropbox API;</>
8. Preview;
9. Publication;
10. Video transcoding.

The following team engaged in the project:

• backend dev (full-time);
• frontend dev (part-time);
• QA (part-time);
• PM;
• BA (part-time);
• DevOps (part-time).

Solving Problem #1

The inconvenience of working with Drupal administration and voluminous content packages

One packet could weigh more than 2.5 GB. We immediately rejected the idea of loading it through a browser on the front-end. The reason was that it could take several hours to download the package; at the same time, you could not close the browser (and what if you accidentally closed it?). To avoid the risks of long downloads and errors, we proposed developing a synchronization with a third-party service installed on Harry's computer. Therefore, uploading to the server does not depend on the browser but can occur in the background.

After the investigation stage, we focused on Dropbox and its external API.

The workflow was as follows:

1. Harry uploads the package to Dropbox into a special folder.
2. In the admin on the package information page, he simply clicks on a "Synchronize" button. The synchronization process starts.

Visually it looks like this — first, he sees the calculation; then, the progress bar shows the download percentage.

1. Once the download process is completed, the status changes. A draft version of the package appears for Harry to review before it becomes available to teachers.
2. If everything is fine, you could transfer that package from the Draft to the Production version; in other words, make it possible for teachers to see the updated methodical material.

Synchronization process

When synchronization started, a list of files was created in Dropbox to be copied to S3. This process looks as follows:

1. The desired Dropbox directory (package) structure is checked. If the next element is a file, then information about the time of change on the Dropbox server is taken from its metadata. This time is compared to the time of the last synchronization in our application.
2. If the change time on the Dropbox server is longer than on our server, then the path to the file is added to the list for copying to S3. During the formation process, the running task processes this list in parallel, copying this file from Dropbox to S3. The key to the S3 storage is the path to the file in Dropbox.
3. After copying is completed, it checks which files are in S3 and not in Dropbox. The list is compiled, and missing files are removed from S3. 
4. The process of transitioning a package from the development version with draft status to production requires additional copying of files in S3 (with a different prefix in the file) and another prefix in the file keys.

After the development stage, we started testing. The testing was supposed to find pitfalls that prevented synchronization—for example, unnecessary or damaged files. The case was considered to be positive if the message about successful synchronization appeared on the screen.

Solving Problem #2.

Video transcoding

The product we were developing is hosted on AWS, so Problem #2 was solved by employing its capabilities.

The content pack contained many video files. Video playback requirements were such that the video had to be played on all browsers and all devices. Previously, our customer had stored videos in packages in two formats: .mp4 and .ogg, which lengthened the time for package formation. We suggested transcoding video using AWS Elemental MediaConvert to streaming video.

Advantages

- when creating the package, Harry had to save the video in one .mp4 format;

- availability of playback in all browsers and on all devices.

How it works

The package contains many static files (HTML pages, CSS and JS files) that reside on AWS S3.

When uploading a package to S3, a manifest with the number of video files in the package is added to the root. Each time a video file is created on AWS, a trigger is fired that increases the counter of created video files.

When the number of video files created in AWS reaches the number declared in the manifest, a job is started in which transcoding is triggered. Transcoding takes some time, depending on the length of the video. In case of success or vice versa, a completion request is sent to the endpoint.

Before uploading the package to S3, a js file is added, which initiates the player for streaming and the necessary changes in the root HTML (usually index.html).

When the player starts, a request is made to the backend; the response contains the desired playlist. It is essential to say that all root HTML should have a similar structure, classes, and element IDs, which simplifies the solution.

The testing phase was directly from the S3 production, meaning AWS did not have a sandbox. We pulled video files from our test server. The testing costs a lot of money for the customer because it is necessary to check how real video files were transcoded, and every second of transcoding has its price.

Technologies:

Python, Django, Celery, Boto3, BeautifulSoup

Results obtained

The project team realized the customer's target, stopping Harry's headache. Now Harry has a convenient admin panel for downloading training content packages, including bulk files. Thanks to the synchronization with Dropbox, educational information uploads on the platform in just a few clicks, quickly, and with the possibility of background download.

The video transcoding function implemented with AWS Elemental MediaConvert allows users to avoid the problem of not playing files of different formats. Thus, video files played in streaming mode work correctly on all browsers and devices.

“Partnership that works for you’ is our guiding principle. We take on complex tasks of custom software development. More than 80 companies have outsourced Software Development Hub to scale their engineering teams and build web applications.

Categories