Implementing Sync & Video Transcoding on Edtech Project

Case of implementation of synchronization and video transcoding on Education domain projects.

We at SDH have already performed plenty of both complex and easy tasks. Sometimes projects have some similar, basic things. Features mentioned in the article are completely unlike others.

• Project represents: 50,000 users. 
• Region: USA.

Description of the task

The customer set us the task to simplify and facilitate uploading educational information packages to the site. The package includes educational content for the teacher, which helps conduct classes with students, a methodical material.

The person responsible for creating the training content (let's call him Harry) has to upload it to the site. By then, the customer already had a Drupal-based admin panel, which was not very convenient. Harry creates a package of HTML pages. These HTML pages have a bright design, full of animations and videos.

Our task was to make that process easier for Harry to easily upload materials for teachers without using FTP and Drupal.
 

Challenges

During the implementation of the project, we solved the following main problems:

• the inconvenience of working with the Drupal admin panel;
• voluminous content packages;
• issues regarding playback of animations and video files of various formats; 
• video transcoding to convert video files into streaming video.

Transcoding is a digital-to-digital conversion of one type of encoded data (video or audio) to another, often because the target device used to display the content requires a smaller file size.

Necessary functionality

It took us more than two months to write the entire project, including sections and functionality:

1. Users;
2. Environments;
3. Packages;
4. Logging of all events in the admin;
5. Access rights in the admin area;
6. Collection of analytics on the user side;
7. Synchronization of the database of the web application and the Dropbox API;
8. Preview;
9. Publication;
10. Video transcoding.

The following team engaged in the project:

• Backend dev (full-time);
• Frontend dev (part-time);
• QA (part-time);
• PM;
• BA (part-time);
• DevOps (part-time).

Solving Problem #1

The inconvenience of working with Drupal administration and voluminous content packages

One packet could weigh more than 2.5 GB. We immediately rejected the idea of loading it through a browser on the front-end. The reason was that it could take several hours to upload the package; simultaneously, you could not close the browser (and what if you accidentally closed it?). We proposed developing a synchronization with a third-party service installed on Harry's computer to avoid the risks of long uploads and errors. Therefore, uploading to the server does not depend on the browser but can occur in the background.

After the investigation stage, we focused on Dropbox and its external API.

The workflow was as follows:

1. Harry uploads the package to Dropbox into a particular folder.
2. In the admin on the package information page, he simply clicks on a "Synchronize" button. The synchronization process starts.
   Visually it looks like this — first, he sees the calculation; then, the progress bar shows the upload percentage.
3. Once the upload process is completed, the status changes. A draft version of the package appears for Harry to review before it becomes available to teachers.
4. If everything is fine, you could transfer that package from the Draft to the Production version; in other words, make it possible for teachers to see the updated methodical material.
    

Synchronization process

When synchronization started, a list of files was created in Dropbox to be copied to S3. This process looks as follows:

1. The desired Dropbox directory (package) structure is checked. If the next element is a file, then information about the time of change on the Dropbox server is taken from its metadata. This time is compared to the last synchronization in our application.
2. If the change time on the Dropbox server is longer than on our server, then the path to the file is added to the list for copying to S3. During the formation process, the running task processes this list in parallel, copying this file from Dropbox to S3. The key to the S3 storage is the path to the file in Dropbox.
3. After copying is completed, it checks which files are in S3 and not in Dropbox. The list is compiled, and missing files are removed from S3. 
4. Transforming a package from the development version with draft status to production requires additional copying of files in S3 (with a different prefix in the file) and another prefix in the file keys.

After the development stage, we started testing. The testing was supposed to find pitfalls that prevented synchronization—for example, unnecessary or damaged files. The case was considered to be positive if the message about successful synchronization appeared on the screen.

Solving Problem #2.

Video transcoding

The product we were developing is hosted on AWS, so Problem #2 was solved by employing its capabilities.

The content pack contained many video files. Video playback requirements were such that the video had to be played on all browsers and all devices. Previously, our customer had stored videos in packages in two formats: .mp4 and .ogg, which lengthened the time for package formation. We suggested transcoding video using AWS Elemental MediaConvert to streaming video:
 

Advantages

- when creating the package, Harry had to save the video in one .mp4 format;

- availability of playback in all browsers and on all devices.

How it works

The package contains many static files (HTML pages, CSS, and JS files) that reside on AWS S3.

When uploading a package to S3, a manifest with the number of video files in the package is added to the root. Each time a video file is created on AWS, a trigger is fired that increases the counter of created video files.

When the number of video files created in AWS reaches the number declared in the manifest, a job is started in which transcoding is triggered. Transcoding takes some time, depending on the length of the video. In case of success or vice versa, a completion request is sent to the endpoint.

Before uploading the package to S3, a js file is added, which initiates the player for streaming and the necessary changes in the root HTML (usually index.html).

When the player starts, a request is made to the backend; the response contains the desired playlist. It is essential to say that all root HTML should have a similar structure, classes, and element IDs, which simplifies the solution.

The testing phase was directly from the S3 production, meaning AWS did not have a sandbox. We pulled video files from our test server. The testing costs a lot of money for the customer because it is necessary to check how real video files were transcoded, and every second of transcoding has its price.

Technologies we used:

Python, Django, Celery, Boto3, BeautifulSoup

Read also: Top Education App Design Trends in 2023

Results obtained

The project team realized the customer's target, stopping Harry's headache. Harry has a user-friendly admin panel for uploading training content packages, including bulk files. Thanks to the synchronization with Dropbox, educational information uploads on the platform in just a few clicks, quickly, and with the possibility of background uploading.

The video transcoding function implemented with AWS Elemental MediaConvert allows users to avoid the problem of not playing files of different formats. Thus, video files played in streaming mode work correctly on all browsers and devices.

'Partnership that works for you' is our guiding principle. We take on complex tasks of custom software development. Over 80 companies have outsourced Software Development Hub to scale their engineering teams and build web applications.
 

Categories