Be careful mixing WordPress optimization plugins

Gears of different shades of blue are connected together showing business related icons.

I spent this morning cleaning up the blog from an issue that was caused by a misconfiguration of WordPress optimization plugins. Specifically a plugin was mixed with the popular WordPress Cache Plugin W3 Total Cache. The actual impact was my fault entirely for ignoring error messages and not restoring a backup right away. This post is a summary of the issue and how I fixed it.

How Wordpress optimization plugins broke my website

After enabling the S3 Compatible CDN Host option to host my WordPress Media and Assets on DreamObjects, a nightmarish situation occurred. All links were downloaded and put into the WordPress Media Folder. I believe that another plugin I used to minify assets must of confused W3 Total Cache’s crawler or something. I ignored the error messages in CDN logs telling me it was uploading links or something. I was tired at the time and made bad technical decisions as a result. Because of my mistake content had to be repaired and a cleanup was in order.

Solution

For most posts the restore process was just go to the WordPress Revisions list and restore the previous version. Since WordPress keeps track of post revisions it wasn’t the end of the world for me but fixing this did eat up a bit of my time. This only worked for about 15 of 19 posts. For the other posts I had to go in and use Google’s web cache to figure out which links to fix and remove anything else which isn’t an ideal solution. To complicate things I’ll still have to go through older posts and add more links where I feel it’s relevant. An outbound link review is generally beneficial to readers and such anyways however the circumstances were unfortunate.

Update: This afternoon someone recommended to me for the old posts that lost the heavy amount of links, I should restore one of my older backups to a subdomain and copy the posts over in source code mode. This suggestion worked perfectly and older posts have been successfully recovered. I’m surprised I didn’t think of this earlier. The backups I had created worked out afterall. Although I still believe that my conclusion stands - be more careful!

Conclusion

I learned the following from this experience.

  1. Test all configuration changes on your staging website first.
  2. Make sure nothing broke right after you change a configuration. I would’ve been able to to just restore an old hourly backup however since then other content has been written and it’s not that simple any more. I keep hourly database backups, each one being only around 30MB is very cheap to store on DreamObjects. In my case I waited too long to test everything (I should’ve paid attention to those error messages!) and it made things harder than it had to be.
  3. Be careful about which plugins you mix and research whether they are compatible with each-other first.

Mozilla's implementation of DNS over HTTPS in Firefox and their claims are misleading at best...

Several copies of the Firefox logo are spread across a square.

Recently Mozilla finalized their implementation of DNS over HTTPS in Firefox. This protocol would improve internet users’ privacy and security while using Firefox. A change in their implementation means that many of the privacy and security benefits of DNS over HTTPS go away for Firefox users.

A history of DNS over HTTPS

March 2018: Mozilla begins testing an implementation of DNS over HTTPS.

October 2018: RFC8484 was published by the IETF to describe the encrypted DNS System known as DNS over HTTPS.

November 2019: Microsoft announced that they would add support for DNS over HTTPS to the Windows 10 operating system.

February 2020: Mozilla announced the inclusion of DNS over HTTPS in Firefox to the general public and began the rollout.

What’s the issue with DNS over HTTPS?

A criticism of the DNS over HTTPS protocol is that it will break some software products that rely on the information sent in a DNS query to determine whether to allow or block a connection to a website. This affects certain educational institutions, corporations, and totalitarian governments.

Popular web filtering products will check each DNS query against an allow or deny list of websites. An encrypted protocol would break those products in their current form and prevent them from working properly.

The solution is to disable DNS over HTTPS on computers owned by the company. Any company computer could simply install a Firefox Enterprise Policy to disable the system while not affecting personal computers which are owned by private individuals. Mozilla’s finalized solution astounded me and goes against what I thought their values were.

What solution did Mozilla provide to network owners?

Mozilla added a simple test to decide whether to allow DNS over HTTPS. If an unencrypted query to use-application-dns.net returns NXDOMAIN or SERVFAIL then Firefox will disable the DNS over HTTPS system. Mozilla had the following to say on their support website about how this works:

In addition, Firefox will check for certain functions that might be affected if DoH is enabled, including:

* Are parental controls enabled?
* Is the default DNS server filtering potentially malicious content?
* Is the device managed by an organization that might have a special DNS configuration?

If any of these tests determine that DoH might interfere with the function, DoH will not be enabled. These tests will run every time the device connects to a different network.

Mozilla Support https://support.mozilla.org/en-US/kb/firefox-dns-over-https

I hold issue with this approach. As far as I am aware, the user is not informed when DNS over HTTPS is disabled. This may give them a dangerous false sense of security. To add insult to injury, they are not offered a way to use DNS over HTTPS against a network owner’s wishes. I was unable to find an option under network.trr in about:config settings to toggle the test. I did notice that in about:studies there is a DNS over HTTPS US Rollout study. Disabling this study might disable the test of whether to disable DNS over HTTPS. If this works, it is a temporary solution at best. Aside from compiling your own “fork” of Mozilla Firefox, it looks like you are forced to obey their decision. Since when did Mozilla get in the business of taking away the freedom of choice from internet users? I thought that was the job of giant corporations, not the non-profits which are supposed to be on your side.

What should Mozilla have done instead?

I believe that this issue could of been easily resolved by adding an option to Firefox Enterprise Profiles to disable the functionality. This would allow normal users to keep using and benefiting from DNS over HTTPS while corporate computers could be monitored. It is the most reasonable compromise and doesn’t undermine the privacy and security rights of Mozilla users.

How does Mozilla’s solution to corporate network owners affect the average internet user?

The solution Mozilla offered to corporate network owners feels draconian and has potentially chilling effects.

Any ISP or Government on demand could return NXDOMAIN or SERVFAIL to disable DNS over HTTPS. This could be used to target specific users (for example activists) by disabling the additional privacy & security benefits DNS over HTTPS offers them.

Anyone with the ability to intercept wireless network traffic could abuse this solution to disable Firefox’s DNS over HTTPS system, then continue the activities that internet users would otherwise be protected from.

Users are not given a warning message that their traffic may be tampered or spied on like they are if an HTTPS connection is tampered with. This goes against the premises of encrypting DNS queries. What is Mozilla doing about this?

Can an ISP disable DNS over HTTPS and continue selling your data?

It is unclear how much data Mozilla is collecting through their rollout study. If major ISPs choose to return NXDOMAIN or SERVFAIL on queries to use-application-dns.net will Mozilla backtrack on their decision to allow DNS over HTTPS to be disabled by a network administrator? As net neutrality is no longer the law, there is nothing stopping them if they choose to do so.

From a technical standpoint, it currently looks like the answer is yes. Allowing ISPs to do disable the system can make it easier for them to sell your web browsing history. It is unclear if the ISPs will choose to override consumer choice.

I can imagine similar situations with a totalitarian government who uses DNS monitoring and tampering to censor the populous by ordering ISPs to block queries to use-application-dns.net once this rolls outside the United States. If they have not taken proactive measures already.

From an ethical standpoint, will Mozilla do the right thing and backtrack once this becomes an issue?

You cannot make a security feature secure unless it protects all users unconditionally

Growing up I was and still am a very active user in information security and privacy technology communities. If there is one thing at all I have learned as a result of those experiences, it is that you cannot make a security feature secure unless it protects everyone unconditionally.

Mozilla’s implementation of DNS over HTTPS locks traffic from otherwise prying eyes but then publishes the master key allowing any entity to unlock the traffic at will. These actions may have chilling effects.

Imagine if the Tor Project modified Tor, an anti-censorship product, to allow easy blocking of connections to the network and stopped providing bridges. It would affect journalists and political dissidents around the world.

What other solutions exist?

Honestly, it hurts me to have to answer this question. I care about Mozilla, and the Firefox Community as a whole. I wouldn’t want anything to happen to it. If anything, I feel betrayed as a Firefox user and speaking out is the only way I believe change will occur.

There is not an easy replacement at this time. The closest thing I found was cloudflared, a command-line DNS over HTTPS client. As far as I’m aware, it does not disable itself to appease network administrators. If you are feeling up to the challenge, Cloudflare provides instructions to configure it.

Conclusion

I do not trust Mozilla’s implementation of the DNS over HTTPS protocol anymore. I was once a strong advocate for it and thought it would improve the internet for the better. Because of their implementation change I can no longer recommend Mozilla’s implementation of DNS over HTTPS. I feel disappointed and heartbroken because of their decision. What I thought would be Mozilla fighting alongside the Tor Project to stop censorship turned out to be false. I can only hope that Mozilla will change their decision and do what’s best for the Firefox Community.

How to automate your own backups with rclone and crontab on any Unix/Linux based computer

Data is backed up to a tape

I’ve been migrating away from Google’s cloud based software. I have concerns related to the security of my data as well as want access to my documents when the Google Cloud or my internet connection is having issues. I was able to download all of my data from Google Drive easily although this creates a new problem. I’m now responsible again for my own backups. Without a backups solution you risk losing your important documents. This post discusses how I created backups with rclone and how you can do the same.

This tutorial is written with Unix/Linux based computers only in mind. You might be able to get this working on Windows if you do your own research and experiments. This tutorial is not intended for Windows users and I cannot help them once something goes wrong.

Where to store the backups

I performed considerable research as to where to store my backups and decided to choose Backblaze. They provide 10GB of storage for free and then charge just $0.005/GB/month of storage. With a provider choose I installed a free and open source program called rclone. rclone works like rsync except for cloud storage providers. I was able to get started with it in just 15 minutes.

Create a Backblaze account and bucket for use with backups with rclone

To create a Backblaze account visit their signup page. You will need to answer a few questions about yourself and provide an email address, phone number, and a credit card for billing purposes. Once your account is set up and verified, visit the my account area. You’ll need to click create a bucket and choose a name for it. Make sure that the privacy setting is set to “private” or anyone who guesses the bucket name will be able to list and download your documents without needing a valid app key with access to your account. Once your bucket is created visit the App Keys section and create a key with access to your bucket. Write down the key id and secret, you’ll need it later when setting up rclone.

Install and Configure rclone to use your bucket

Visit rclone’s installation page (or on Debian Linux type sudo apt install rclone in a terminal window to save some time) and follow the instructions. Once it’s installed open a terminal. In the terminal run rclone config. It will ask you several questions, I choose the name b2 and then follow the prompts. I would recommend that you DO NOT enable the HARD DELETE option and allow your bucket to keep all object versions. It’s a bit more expensive as you’ll store multiple copies of all documents but is useful in the event you delete a document by mistake and the changes were to sync in error.

Configure crontab for automated backups with rclone

In your terminal type crontab -e and then view the bottom of the crontab file. Type in the string @hourly rclone sync /home/name/Documents/ b2:name/ --verbose replacing the folder path with your documents folder path and the bucket name with your bucket name. Most likely the crontab file will open in nano. You can read how to use the text editor nano on the Gentoo Linux Wiki. If it opens a different text editor then Google is your friend. It will help you find the information you need to use that text editor. Afterward save and close the file and your automatic backups will be in effect. You might want to run your first backup manually. Do this to make sure everything gets synced properly without any issue. Check your backups on a regular basis and ensure that they work as expected. Don’t wait until a data loss incident to find out if your backups work.

Conclusion

You can reduce costs by switching away from G Suite to your own backups solution. With a bit of work on the command line, you can roll your own backups with rclone solution in under an hour.

10 Years of Debian (and Ubuntu) Linux...

This post is a reflection on at least the past 10 years of my life, how Debian fits into it all. This post goes into a lot of personal issues (including mental health) so if those topics are sensitive to you, perhaps skip this post. With that said our story begins…

How I discovered and installed Linux for the first time

I started using Linux sometime between the ages of eight to ten years old, this is an estimate as I don’t have exact dates only my memories of the events played out. My primary computer at the time was an HP netbook style computer. The netbook came installed with Windows 7 Starter Edition, a somewhat downgraded version of Windows 7. This was common for netbooks at the time. The netbook was small, light, and packed a punch for the time.

Many of my issues with the netbook getting slower probably came from installing sketchy programs. I was into the online video game Club Penguin and had installed a random program that could trick one of the mini-games into giving me unlimited coins, it was pretty cool. Choices like that probably contributed to slowing down my netbook.

My cheating in online video games turned out to be rewarding in ways I wouldn’t understand until much later on in life and lead to new unique discoveries. Even if not under the best circumstances I’m thankful for these experiences as things turned out okay.

I read some blog posts about how Linux could make old computers faster. I became curious as to what Linux is and how it’s different from Windows. As a result, I started looking into how to install Linux and discovered my first and still current distribution of Linux, named Ubuntu. Ubuntu is Debian Linux with user-friendly features added on top along with commercial hardware drivers built into the image and available out of the box. There is some controversy that Ubuntu is not completely free and open-source software because of this change, however, it is a nice mix for the average computer user. The version I started with was Ubuntu 11.04 (Natty Narwhal), I remember this because it was one of the first versions (Ubuntu netbook edition already had Unity before Ubuntu 11.04) of Ubuntu to use the Unity User Interface).

What has changed since April of 2011

Quite a few things have changed over the past many years. The most important thing to me is I’m a lot more comfortable using the command line making me a more effective Linux user. You can use the command apt instead of apt-get to install software (see apt vs apt-get). Ubuntu has made significant progress on its user interface since then. They switched from their custom user interface Unity back to Gnome3 and added their own unique tweaks and elements on top of it. This allows them to take advantage of Gnome’s development team and let them work out the hard part while they focus on designing a better user interface and building it on top of Gnome3.

Debian and Ubuntu have kept up with the latest Linux kernel releases and has continued to support exponentially more difficult hardware due to changing vendor requirements. They both continue to make their operating systems as a whole more secure.

Ubuntu added a new software deployment system called Snappy) which adds further isolation to software and automatically keeps it up to date. No more clicking “install update” it just works in the background.

Canonical), the developers of Ubuntu, is a for-profit business that makes money by selling software support services to enterprises. They added a software component called LivePatch to remove the need to reboot to apply critical security updates. It’s an optional service and is not enabled by default.

I became interested in hosting websites and managing virtual private servers

Over time as my interest in Linux grew, I became interested in making simple websites and managing servers. All of these servers ran Debian Linux. My laptop and server management skills both grew over time. I remain super comfortable with the command line and Debian’s unique features and quirks.

I started learning Linux systems administration by learning about Apache and Nginx configuration. Until the end of high school, I preferred Apache. The configuration syntax of Nginx was overwhelming. Over time I overcame this struggle. Today, Nginx’s high speed and advanced configuration are essential tools for developing and deploying web applications.

I started out programming with PHP. WordPress is written in it so it’s what I started with. Over time I learned JavaScript and Ruby on Rails. My favorite database is PostgreSQL although I still use MySQL and Redis for various tasks. I learned the Crystal programming language when working on my first big project from my current employer Universal Layer LLC. I made a lot of friends in its community and its Ruby-like syntax helped me remain productive.

A brief history of my Linux Distribution Usage over time

Throughout middle school, my primary Linux distribution was Ubuntu Linux. In high school, I had begun seriously struggling with anxiety and depression. A side effect of the mental health issues and medicines was that I became super paranoid.

When you mix these issues with someone with an interest in information security things get interesting.

To protect myself I used a Linux distribution called Backbox Linux. Backbox is a Linux distribution designed for security researchers. It is based on Ubuntu and has built-in full disk encryption. Backbox Linux has a built-in stealth mode. This mode enables special security features and forces all network traffic through the Tor network to hide your identity.

Backbox was a daily driver for me. I learned a lot of cybersecurity, in a mostly negative way, as a result of these issues. Some may say I learned the hard way but with skills that will last a lifetime. This was arguably way more security than I actually needed but it served it’s purpose and served it well.

Since then I’ve moved back to standard Ubuntu Linux on my computers.

What I currently like about Debian Linux

Aside from the my personal history of how I came to use Debian Linux, let’s take a look at what I like about the operating system. The most upfront benefit of Debian Linux is it’s strong security and speed. Debian runs very few programs in the background by default. This makes it very secure as well as fast. It provides a peace of mind as I continue to work through out the day. I like the software Mozilla Firefox and LibreOffice that is bundled with it, and there are a variety of tools available for various programming tasks.

Do I prefer Debian or Ubuntu

This is somewhat of a trick question as Ubuntu is based on Debian. I prefer Ubuntu on my personal computers due to their opinionated user-friendliness tweaks. I also appreciate the amount of commercial driver support Canonical added to it. In order to remain a productive software developer I have to make compromises between freedom and productivity. I lean towards greater productivity. Canonical is working hard to improve the enterprise adoption of Ubuntu. With the Ubuntu Advantage program, it’s easier for large enterprises to adopt Ubuntu. I imagine we will see more computers ship with Ubuntu by default. As more applications are turned into websites, needing Windows or macOS will be less necessary to access commercial software.

Here’s to the next 10 years

For better or for worse I know Debian and Ubuntu will continue to be around. They will improve my life and computer use as a whole. I can’t wait to see what happens next.

Will Google ever run out of storage space for YouTube?

A few years ago I wrote an answer to this question on Quora which says no. I wanted to revisit and reflect upon this topic and on YouTube as a whole. This post is my updated thoughts on the question.

YouTube is changing…

Since 2017 Google has continuously updated the platform to comply with the law and to keep their advertisers happy. Without advertiser money or charging for the service (they have a few paid consider products, YouTube Premium and YouTube Music), there is no YouTube. As the platform changes, so will their technical requirements, with access to the smartest software engineers on the planet I trust that YouTube will find a solution to the challenges they face.

The rise of NVMe Storage and UDH Video…

It’s 2020 and NVMe storage is more accessible than ever, UHD Video (4K & 8K Standards) is becoming more popular than 1080p video. As storage needs skyrocket, YouTube will have to adapt to growing storage requirements. If ad revenue drops will they have to start charging creators a storage fee? It’s unclear however considering the massive scale of YouTube and that they can display ads on any video I do not think they’ll start charging creators fees.

New data compression algorithms…

As Cloudflare and many other content delivery network companies have shown there is no shortage of new data compression algorithms and techniques. I’m sure somewhere behind the scenes at Google, magic is being put together to make 4K and 8K video files smaller than ever.

My answer on Quora in 2017

I’ve quoted my answer below for reference. I’ve pulled my answers from the Quora platform recently as I want to focus on blogging so you’ll have to read them here instead.

“Can’t just keep adding more storage.” That’s exactly what they’ll do. Consider the following:
1. Google has a lot of money. They can spend as much as they need to add more storage. If needed they’ll build more data centers. Whatever it takes to keep running YouTube.

2. Hard drives are being able to store more and more data. They’ll upgrade as needed.

3. Google has some of the smartest engineers on the planet. I wouldn’t be surprised if they had proprietary compression technologies to make the most of their hard drives.

Hope this helps :)

Nathaniel Suchy on Quora - June 6th, 2017

My review of the Lenovo ThinkPad X1 Carbon 3rd Generation

Recently I got the Lenovo ThinkPad X1 Carbon 3rd Generation laptop for some light C#/ASP programming for work. This post is on my thoughts on the unit, it’s performance, and whether it’s right for you.

Build Quality

It’s a pretty sturdy laptop although a bit thin. I feel a bit on edge about opening and closing it, nothing has happened so far although I’m afraid I’ll use too much pressure and crack it. At some point laptops become too thin. I’m not sure it reaches that threshold yet. I’m the upside it’s lightweight and easy to type while walking so that’s a plus at least. Maybe I am not used to such a thin and lightweight laptop (the MacBook Pro is large and heavy).

Display

My view is arguably tainted as I bought the laptop used, but the display is hard to view at an angle. I’ll be purchasing and installing an IPS display in the near future. I’ll update this section after I install it. Right now I have to view the screen at a specific angel or it’s hard to read. I was able to watch a movie last night without any issues so I imagine there will just be an adjustment period.

Performance

The unit features an Intel i7-5600U and is very fast. There is an 8GB of RAM limitation but it packs a punch at a reasonable price range.

Keyboard

This is by far the most comfortable typing experience I’ve had so far. I have yet to see a better keyboard on a laptop.

Trackpad

Admittedly the trackpad was a bit tricky to learn compared to the flat square mousepad on my Macbook Pro but overrall it served it’s purpose and I prefer to use keyboard shortcuts wherever possible anyways as it’s faster.

Storage

The unit features an M.2 SSD and is very fast although it’s limited to 256 GB of storage. I tend to hoard a lot of data and will either need to upgrade the internal disk or set up a network file share in my bedroom network I’m not sure which yet got a check how much storage costs about $400 per 16 TB according to Amazon.com.

Conclusion

I am currently enjoying the ThinkPad and this may become my primary laptop. As a Mac user learning to use it and Windows 10 Pro has been difficult but I’ve enjoyed it so far and would recommend it to anyone looking for a laptop on a budget.

Ruby 2.7 is released to the general public

Recently the Ruby Project announced that they are releasing Ruby 2.7.0 to the general public. This post is intended to summarize the changes to a non-technical audience.

Summary with Personal Insights

The Ruby 2.7.0 update adds: pattern matching, an improved REPL (command line tool to write ruby in), improved garbage collector (reduces memory usage and improves efficiency), and separates positional and keyword arguments.

The new pattern matching functionality rivals that of Elixir’s pattern matching and will provide new problem solving tools to Ruby Developers.

The REPL is using a new version of readline, it’s faster and now has syntax highlighting. This will make things more readable and provide a more pleasant experience to Ruby developers. Maybe improved to the point where third party gems (libraries) like PRY are no longer necessary except for specialized debugging workflows.

Improved garbage collection will speed up run-time performance and be extra helpful to developers running on older hardware and in memory constrained environments. The user can directly invoke it however they should proceed with cautious and work to write memory efficient code from the start rather than cleaning messy memory later on.

The separation of positional and keyword arguments is a bit tricky to explain. Imagine you have a method which asks you to explicitly declare keywords and pass a value. Previously you could put this information into a Hash (a type of dictionary like data structure) and it would be automatically converted. This behavior is error prone and deprecated as a Ruby 2.7.0, to request a conversion in the future you’ll need to prepend a hash variable with **variable_name to have a positional argument converted into the required keyword argument. It’s my understanding that eventually this behavior won’t be allowed at all so it’s best to update your code now rather than later.

Source: https://www.ruby-lang.org/en/news/2019/12/25/ruby-2-7-0-released/

How I created TwitterPub and why you should use it

Recently I started working on a project named TwitterPub. The goal of the project was to allow you to follow and interact with Twitter accounts over Mastodon and other ActivityPub software. It’s still very much a work in progress and doesn’t completely work yet. This blog post is to summarize what I learned as well as a progress report of what parts I’m still working on.

Pulling data from Twitter without their API

A while back Twitter suspended my personal account without providing a reason. They are within their rights to do this although I wish I had been given a reason. They denied my appeal as well. Since I am not allowed to have a Twitter account anymore I had to get creative with how I pull data from their platform. If there’s a will then there’s well me. Thankfully Twitter still lets logged out users view public profiles and tweets and that’s enough for me. I was able to have Ruby pull and parse this information. To prevent Twitter blocking the requests later on, I sent falsified HTTP Headers to pretend to be Firefox 71 running on macOS Catalina the requests look legit to me ?

Pulling and parsing a Twitter Profile

To pull and phrase a Twitter profile without their API I used HTTParty and Nokogiri to pull and phrase the pages. I selected elements by their CSS class. Due to how Twitter generates its pages it made it relatively straightforward to pull data from a profile. The drawback is that if they change classnames the code will have to be updated and they could quite easily break TwitterPub if they decided to do so.

# Get Profile Object from a Twitter Profile URL
def get_twitter_profile(url)
    # Get page and parse with Nokogiri
    page = HTTParty.get(url, {
      headers: {
        "User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0",
        "Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language" => "en-US,en;q=0.5",
        "Cache-Control" => "max-age=0",
        "Connection" => "keep-alive",
        "DNT" => "1",
        "Upgrade-Insecure-Requests" => "1",
        }
      }
    )
    parsed_page = Nokogiri::HTML(page)

    # Assign data from page to variables
    display_name = parsed_page
                    .css('.ProfileHeaderCard-nameLink')
                    .inner_text

    account_name = parsed_page
                    .css('.username .u-linkComplex-target')[0]
                    .inner_text

    bio = parsed_page
            .css('.ProfileHeaderCard-bio')
            .inner_text

    website_link = parsed_page
                    .css('.ProfileHeaderCard-urlText')
                    .inner_text
                    .gsub("\n", "")
                    .gsub(" ", "")

    location = parsed_page
                .css('.ProfileHeaderCard-locationText')
                .inner_text
                .gsub("\n", "")
                .gsub("              ", "")
                .gsub("  ", "")

    join_date = parsed_page
                  .css('.ProfileHeaderCard-joinDateText')
                  .inner_text

    avatar_url = parsed_page
                  .css('.ProfileAvatar-image')
                  .attr('src')
                  .value

    header_url = parsed_page
                  .css('.ProfileCanopy-headerBg')
                  .children[1]
                  .attr("src")

    # Create and return a profile object
    profile = Profile.new(display_name, account_name, bio, website_link, location, join_date, avatar_url, header_url)
    return profile
  end

Pulling and parsing a Twitter Status

I used the same approach except less specific parsing was required for statuses.

# Get Tweet Object from a Twitter Status URL
def get_twitter_status(url)
    puts url
    # Get page and parse with Nokogiri
    page = HTTParty.get(url, {
      headers: {
        "User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0",
        "Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language" => "en-US,en;q=0.5",
        "Cache-Control" => "max-age=0",
        "Connection" => "keep-alive",
        "DNT" => "1",
        "Upgrade-Insecure-Requests" => "1",
        }
      }
    )
    parsed_page = Nokogiri::HTML(page)

    # Assign data from page to variables
    status = parsed_page
              .css('.tweet-text')
              .inner_text

    timestamp = parsed_page
                  .css('.metadata')
                  .children[1]
                  .inner_text

    # Create and return a status object
    status = Tweet.new(status, timestamp)
    return status
  end

How Twitter could stop me from doing this

I think the easiest way by technical means would be to randomize the CSS class-names. I would then have to enumerate through the DOM in another more difficult way. Twitter could also add some “checking your browser” style screen but it may interfere with the mobile apps. They could also require an account to view public profiles and tweets (then whitelist Google and other search engines). Lastly Twitter could have their corporate lawyers send a cease and desist letter and use the legal route. I don’t think they care but I’ve read they do block IP Addresses which hit too many 404s. I guess I’ll see what happens in practice.

ActivityPub is complicated

Turning this data into ActivityPub responses is more difficult. I had to implement a lot of different actors and then an outbox. To properly implement an outbox (required to display posts on a profile) you have to store copies of the profile and put into a complicated JSON response. I’m still working on using sqlite to do this. I’m also looking at what I’d need to do to copy every tweet from a profile and remove duplicates with close to 100% accuracy. It’s very much a work in progress but I hope to do more soon.

What’s next

I want to add the ability to reply to tweets over mastodon. I’m still researching ways to go about this. I will likely have to implement parts of this with Twitter’s API for a third party to run their own relay if interested. It’s the only way to reply to Twitter accounts that I’m aware of. I want to get a working demo out there as well. I hope you found this post insightful and are interested in seeing the completion of TwitterPub.

Introducing me

Hi I’m Nathaniel Suchy: A Software Developer Consultant at Universal Layer LLC, and a Student at Wake Technical Community College. I am finishing my degree program for an Associate’s Degree in Applied Science for Web Development.

My programming background is Ruby an object oriented interpreted language which tries to have syntax that looks similar to English. This makes it easier for people to read and write it. The English-like syntax is helpful to show business logic to a non-technical audience.

After I graduate Wake Tech I am looking for a entry level software engineering position at a mid sized company.

My thoughts on Amazon Device Financing

One of my goals in 2020 is to read more books (I read a large amount of blog articles and wikis every day by no means have I skipped reading after I graduated high school). I looked at Amazon’s new Kindles and decided to order one. I have not used e-readers in a very long time and decided to give them another chance. I noticed at checkout that I was given the option to pay in five monthly interest free installments of $18. This was a friendly bonus as I was about to spend $100 either way and Amazon just made my night much easier. It was even better when I was not asked for my social security number to pull a credit report.

Eligibility & Requirements

I imagine that Amazon’s algorithm decides who it’s safe to loan money to based on several factors. And they probably won’t tell us everything to stop people from gaming the system. As in their deciding factors for me personally, it probably helps that my household as a whole spends several hundred dollars a month on various Amazon purchases without really thinking about it. We procure so many various tiny things and rely on Amazon to send them on time. Reflecting on my use of Amazon as a whole, the world would really be different without such an easy online shopping platform. We were rearranging the kitchen today and we needed more surge protectors because we moved the microwave to another part of the kitchen luckily it took all of one minute to open the Amazon app click purchase and confirm the order and it’ll be here tomorrow afternoon. Imagine having to go back to the store for the same product waiting in the checkout lines hoping that they have a surge protector available, let alone available at the same price. It would really change the way our workflows went and slow down life. The only limiting factor in my purchase was my Wi-Fi speed. Back to discussion of Amazon’s special financing their offer page states:

“To be eligible for this offer, you must reside in the United States of America, your Amazon.com account must have been active for at least one year, you must have a valid credit card associated with your Amazon.com account, and you must have a good payment history on Amazon.com. This offer may not be available to every customer and may not be available to you for all qualifying products. We reserve the right to consider for each transaction factors including your transaction history and past products purchased on Amazon.com and the nature and price of the qualifying product, in determining your eligibility for this offer. We will not use a credit report to determine your eligibility.” https://www.amazon.com/gp/help/customer/display.html?nodeId=202095050

Instead of a credit report they’re basing it on your payment history with Amazon. And the fact that you have a valid credit card attached to your account. This offer is extra nice given that while I am not eligible for Amazon’s Prime Visa Card due to limited personal income, I was still eligible for a secured credit card from Discover Bank and it helped me meet the eligibility requirements of this offer.

Benefits to Amazon

Most of Amazon’s customers make several purchases a month and create constant cash flow for the company. By giving customers the device early and on a payment plan they increase the odds the customers will stay subscribed to Amazon Prime (for Prime Reading) and Kindle Unlimited and that they’ll buy more books on the Kindle platform. Amazon doesn’t have to wait for the customer to decide they’re willing to spend a lot of money at once and they increase the odds that the customer will purchase a Kindle and bring all of the additional cashflow that comes with it. There is always the risk a borrower won’t pay back but because of DRM Amazon can disable the devices remotely, disincentivizing people from deciding to stop paying. And Amazon still has a lot of data on who will and will not pay on time.

Checkout Process

It was a fast process done from the iPhone app. I got this information attached to my order: I had to pay $24.07 upfront and still owe them $71.99 over the next few months in payments of $18.00. Amazon is risking $71.99 in a worse case scenario. They don’t lose this money unless a customer fails to make the required payments. The device sits in a warehouse until it’s ready to be shipped. If all goes according to plan Amazon continues to make a profit from all of the subscription payments and e-book purchases throughout the lifetime of the product. This is a really smart move on Amazon’s part and I imagine it must be paying off pretty well for such a large company.

Conclusion

Amazon makes it easier for me to justify a large purchase knowing that I lose the money over time and not all at once and have the money available to cover other expenses and increasing both of our cashflow. Yay capitalism?!?