Week 10 & Conclusion

With the research completed, the final presentations over, and the start of a new school year fast approaching, it’s time to reflect on the ups and downs of this summer and set new goals moving forward.

First, I would just like to say how happy I was with the program as a whole. I was not expecting to make the sort of progress that I did, and was able to come away not just with a good research experience, but also with a solid deliverable — and more than enough substantive material to fill a two minute elevator speech. If you had told me in April that I would be able to answer the “What did you do this summer?” question with “I built a completely autonomous process for a 3-D printer to accept new jobs via email,” I wouldn’t have believed it. Ten weeks ago, the complexity of automation was barely on my radar (I wrote it off with one sentence in my first blog post — “Week 6: Add wireless capability to FL-Sun7 3D. This can easily be done using a Raspberry Pi and the open source software Octoprint”), and I had been more focused on improving the temperature sensors on my 3D printer. While this could still be in the cards for the future, Jonathan’s work this summer is what really allows high quality prints, and it seems to me that my process will serve best for rapid, remote prototyping of ideas. In fact, Jonathan, Ryan, and I’s work all proved to be much more interconnected than we all had initially thought. Our research this summer laid the groundwork for a future in which someone could, with minimal effort, use a drone to capture and create a 3D model of any subject (Ryan’s work), rapidly prototype a print of that model remotely (my work), and then print it later as a high quality resin model (Jonathan’s work). The exact workflow for this is still a little ways off, but it was awesome to see the flashes of brilliance from everyone in the group that signified how close we are to something like that.

We also learned the importance of remaining flexible, yet motivated. The focus of my project might have changed three or four times throughout these ten weeks, but I never at any point felt that I wasn’t making positive progress. Proper documentation, as practiced in things such as these blog posts, serves to prove to oneself just how much as been tried, even if not that much as been accomplished. We were able to easily hold each other accountable, and exercises such as the biweekly lightning rounds with the DSSF scholars helped to serve as benchmarks for our progress. The occasional joint workshop, while not always catering specifically to our needs, also helped to mix things up and prevent monotony.

Most importantly, we gained experience conducting independent research. I never felt as though I was being watched during this program, but instead I was decidedly aware of the wealth of resources available to me should I need them. Upon encountering a problem, the process usually went like this: First, Google (duh). Then, ask coworkers or our friends in IT. If that didn’t work, we always knew we could go to Eric or Sharon and receive further guidance. It is impossible to understate the importance of being able to solve your own problems.

On that note, I should probably take note of what I accomplished in my final week. I spent a long time trying to help Jonathan set up his new kit (which I’m sure he will go over himself), and we also set up a dedicated Raspberry Pi for him to test configurations on PLA prints. In my research, I did manage to implement the solution to a problem I concluded with in my last blog post — allowing the user to cancel a job mid-print — although not in the way I had originally hoped. Due to the fact that I only had one touch screen device, I decided to change my application to work primarily with a keyboard for ease of use across devices as the program came to a close. The user-program interaction now goes as follows:

  1. The user sends an email, containing a .stl file, to the email address I have set up.
  2. The handler program notices a new email, and checks the sender’s email address and their secret code against the database, also verifying that there is an attachment present of filetype .stl.
  3. After passing the required checks, the handler creates a new job in the database, and assigns the job to a vacant printer.
  4. The node program notices that a print job has been assigned to its own printer id, downloads the correct attachment from the email, slices the print, and starts the job, updating its progress online as it goes.
    1. Should the user wish to cancel the print midway through, they press Control + C on the keyboard connected to the RPi that is processing their print. In the future, this could be done remotely.
    2. If the print is cancelled, the program asks the user if they would like to re-attempt the print. If the answer is no (done through keyboard input),  both the .stl file and the job in the database are disposed of. Otherwise, the print is re-attempted as normal.
  5. If the print continues to completion, the node waits for the user to retrieve the print off of the bed. Once it has been retrieved, the user lets the node know by inputting the correct command using the keyboard.
  6. Once the node registers that the completed print has been removed, it disposes of the .stl file and removes the job from the database, signaling to the handler that it is available to accept a new job.

This process generally works flawlessly, and I was able to demonstrate it fully during my final presentation on Friday! However, there are currently a few drawbacks. Occasionally, the handler or the node briefly loses connection, which causes connections to the mail server or database to fail. I have implemented try/catch blocks to attempt to retry the connection when this happens, but for some reason it doesn’t correctly re-establish.

Continuing Research

I would certainly like to continue working on this project, and I have been presented with the opportunity to do so by using the project as my senior capstone project for Computer Science. Were I to accept this opportunity, I would likely take steps to implement a number of new features into the program, such as:

  • Working with IT to allow a real login process, using a Gettysburg ID
  • Allowing remote upload of files via the website
  • Implementing webcam support to watch prints as they complete
  • Alternative status alerts on prints, such as email or SMS
  • A robust scheduling system to assist with the timing of print jobs
  • Training system to show how to build and maintain the printing setup (already underway)

To conclude, I would just like to repeat how lucky I feel for being selected to this program and how proud I am of my own results and my peers’. This is the beginning, not the end, of my work in this area.

James

Week 9

Approaching the end of the project, and everything is finally coming together.

The first thing I did this week was to redistribute my workflow across multiple Pis in order to allow for better scalability. I now have Pis serving two different kinds of functions, one called a handler, and one called a node.

Handler

While this program will eventually be run off of its own dedicated Pi, it can be run from anything, including my own personal computer. What the handler does is somewhat self-explanatory: It handles the assignment of jobs to each printer on the network. Upon starting, the handler searches the email inbox for new mail that has not been added to the database (a process described earlier — I essentially just copied the code from my previous week) and adds it. If all the necessary security checks pass, it adds a line to the JOBS table in my database, assigning the job to a (imaginary) printer with PRINTER_ID = 0.

Once that is done, the handler retrieves all jobs from the JOBS table with a printer ID of 0 (there may be many, as there could be multiple prints waiting for a free printer). With a list of all available jobs, it then compares the jobs which are currently in progress (they have a nonzero printer ID assigned to them) with the master list of printer ID numbers. So, for example, if the network has four printers with IDs 1, 2, 3, and 4, and the JOBS table looks like this:  

Then the handler will reassign the PRINTER_ID value for the row containing file unassigned.stl to be equal to either 2 or 4, since those are the printers which currently have no jobs assigned to them. Once this is done, the handler continues scanning the mail inbox for new messages, and leaves the rest of the work to the nodes.

Nodes

Instead of scanning the email inbox for new messages, the nodes simply check the JOBS database for jobs which have been assigned to their own PRINTER_ID. Once it finds one, the node fetches the email with the given UID, downloads its attachment (which has already been verified to be of the correct filetype and from a trusted sender by the handler, however the node also rechecks for this information), and uploads it to the OctoPrint instance that it is running locally. Due to the school’s firewall restrictions preventing the generation of temporary API keys (meaning that the only way to guarantee that a print will go through is to actually log into OctoPrint from the Pi’s local network, which is impossible to do on a headless setup not connected to another computer), one of the changes I had to make involving the implementation of the nodes was to switch them over to OctoPi’s GUI setup (the desktop version). While this is less than ideal, because it would be nice to have every Pi running completely without a monitor and on its own, there are solutions to this and in the long run it will help with debugging. I got a (very small) touchscreen monitor for my Pi, and used that to log in to OctoPrint and run my node program.

After the node starts the print, all it has to do is wait until it is finished, collecting updates on the print progress and pushing them to the database, where they are then updated on a website I made:

Once the job reaches 100% completion, it waits for the button to be pressed before removing the job from the database, freeing up its slot as an available printer, and it waits to accept any new job the handler assigns it.

The node and the handler work great together, and the system seems to be just as scalable as I had hoped. One remaining challenge is that I currently have not figured out how to handle a print that is going wrong, i.e. the user wants to cancel it in the middle of the job. This could be remedied by installing a larger monitor, as the user could then use OctoPrint’s interface to cancel the job and the node would simply restart the print. However, this might not be the optimal solution and is something I am going to be exploring in the coming week.

In the future, the webcam stream of the prints would also be shown on the website but I currently am not sure about the viability of the webcam idea given the time constraints I am working under. I think it may be better to spend the last week fixing the problem of a failed print and working out any other bugs that I may find, as well as writing detailed documentation for various parts of my project which may need to be serviced or altered in the future by persons other than myself. My final blog post will contain links to this documentation, as well as summarizing my last progress points and where the project will go from here.

Week 8

This week, my primary goal had been to build a queue which would facilitate not only multiple printers connected to the same database, but also a potentially long list of pending jobs which would need to be delegated to the first available printer at the end of the job’s completion. I had then planned on beginning webcam implementation, but ended up becoming engrossed in a separate issue which consumed most of my week — but not necessarily in a bad way.

The Button

I realized that in order to begin building, and then testing, a queue, I kind of needed to have some existing button setup. This is a simple button attachment from Adafruit, and the code to simply wait for the button to be pressed and perform an action is fairly simple (code used adapted from here):

The Database

I also created a JOBS table in my database which initially only contained a list of mail UIDs, the date the job was added, and the job’s filename. Using this in conjunction with the button, I was able to make it so that when my program detected a new job, it would check to see if there was a current job in progress. If there was not, it would start the job and move on, adding the Job into the database as it went. If there was a current job in progress, but the job was not completed, it would simply add to the database and move on. Finally, if there was a completed job at the printer, it would wait until the button was pressed before starting the next one. Below is the relevant code:

However, I quickly realized that entering an infinite loop like this, especially in a situation where print jobs could be potentially left sitting on the print bed overnight, would not only be impractical and also highly limit scalability to a setup with multiple printers. It took a long time to come up with an adequate solution, but I believe I have found the correct way to do it.

Moving forward

The way I envision this project to work is now as follows. There will be one Raspberry Pi which acts as a sort of central server, scanning the email inbox for new jobs and adding them to the JOBS table as a queue. The JOBS table now contains a column denoting the PRINTER_ID, which is equal to zero if the job has not been assigned anywhere. This central RPi has a list of all PRINTER_IDs, and checks the JOBS table for printers that have not been assigned jobs, delegating them accordingly. I already have this program essentially working as it should, and will be covering it more in next week’s post.

Then, each printer will be connected to its own Raspberry Pi, which will simply periodically connect to the database and check for jobs assigned to it. Once it finds one, it will connect to the email server, download the correct email, and start the job. Once it is done, a user pressing a button next to the printer will signal to the database that the job has been completed, and the print will be removed from the list of active jobs, freeing up the printer to be assigned a new job by the server. This not only more evenly distributes the amount of resources used by each Pi, but also allows each node to only focus on one thing at a time, as a Pi working on completing a print job will have no need to scan for incoming emails or add anything to a database. Next week I will also be beginning work on the forward-facing website for monitoring prints, and will hopefully also get some webcam support up and running. More details to follow.

Weeks 6/7 & Updated Timeline

Combining posts again due to a short Fourth of July week. I’ve made some good progress over the last couple days, and think I have a pretty good idea of what my final result is going to look like.

The Database:

In order to track emails as they are received, sort them according to verified or nonverified senders, and to eventually implement a live website which uses this data and tracks print progress, I needed to create some databases. Below is a picture of a preliminary outline of database schematics and how they would interact with the program:

The current database setup I have is as follows.

ADDR_UID

This table simply stores the UID and sender address of all incoming emails to the inbox, regardless of verification status, etc. Useful for identifying spammers or debugging. UID has a UNIQUE attribute because no two UIDs should be the same.

V_A

A manually-entered list of verified email addresses and secret keys. Management of this will eventually be handled by a website or some other form of authentication, and the keys will eventually be stored as hashes. Currently, the method of secret verification relies on the subject of the email being sent, but this is certainly subject to change to some other form of authentication.

V_M

Verified mail. Stores all email from verified addresses, but the program only opens attachments or pushes a print job if the filetype is ‘.stl’, making debugging and spam filtration possible.

Next week I am going to be working on adding a job queue, with an associated database table, so that after a job is completed, the next one can be automatically started. This will likely require design of some sort of physical button that can be pressed after the user retrieves their print, which in turn triggers the start of the next job.

The Program:

Here’s a rough pseudocode outline of what my current program does:

function main():

  function init():
     initialize IMAP server, login to email, select inbox
     connect to database, initialize database cursor
     get list of email UIDs in inbox

  While True: (this creates an infinite loop)
     Check if most recent mail == most recent mail in database
        If it does: 
           Wait ten seconds and check again
        If it doesn't: 
           run emailHandler()

function emailHandler():
   
  Create HTTP Request Object
  Retrieve email body structure, raw data
  Check email for attachments, parse filenames
  If email has nonempty list of attachments:
     run DBHandler()

function DBHandler():

  Insert message UID, message Sender into table ADDR_UID
  Retrieve list of verified email addresses, secret
  Check to see if message sender is verified, secret code is correct
  If sender verified, message secret correct:
     If filetype is '.stl':
        Download file
        Insert message into V_M with VALID_ATTACHMENT boolean
        run addFile()
     Else:
        Insert message into V_M with INVALID_ATTACHMENT boolean

function addFile()
  
  use HTTP request object to send '.stl' file to OctoPrint
  use HTTP request object to start print job


TODO: Determine if print is currently in progress, set up queue

The skinny of what this means is that, while the program is running, a verified user can send me an email containing an .stl file from anywhere in the world, and my printer in the West Building at Gettysburg College will download the file and begin printing. 

My program uses imaplib, which is a built in Python module which allows it to enter an email inbox and read mail. It also uses MySQLdb for database interaction. Here is a screenshot of my initialization function, which logs in to my email address, and returns either just the mail object or also connects to the database and returns both the mail object and a database object. The variable passed to the function is simply an integer that specifies which process to run.

From there, it checks to see if the ADDR_UID database is current with the mail inbox. If it isn’t, it initializes an HTTP request object courtesy of Requests: HTTP for Humans, and then begins processing the new email. SMTP provides several methods with which to analyze emails, the most common protocol being labelled RFC822. Fetching an email object using this protocol returns a number of objects, which can be used in conjunction with the aptly-named python connector email to retrieve things like the message sender, subject line, body text, and even download attachments. However, since we have some specific requirements as to what our desired email attachments should look like, it would be highly preferable to know some more information about the nature of an email’s attachments, or if it even has attachments, before we begin unpacking its raw email data. This is where the IMAP protocol BODYSTRUCTURE comes in handy. It shows some various metadata about a given email, including its attachments, but is not very readable. Below is a sample BODYSTRUCTURE of an email.

Not the most readable. But, if you look closely, you can see that part of it reads ‘(“ATTACHMENT”‘, ‘(“FILENAME”‘, ‘”h8KE9Rx.jpg”))’.  As it turns out, all the emails I tested contained either this exact format, or the same format with some minor format differences. Thus, by making use of Python’s built in split() function, we can create a list from the above structure, search for instances of strings such as ‘(“FILENAME”‘, and then know that the next index of the list will contain the name of the attachment! Here is the code that I arrived at, which currently has passed all tests I have thrown at it (from email addresses on Gmail and Outlook):

These tools and functions allowed me to read every email in a given inbox and return a dictionary containing various information about each email that has an attachment, like so:

This are simply the details from the two D3 textbooks I downloaded from Leanpub.

This was one of the hardest parts of the whole process, since there wasn’t much guidance online for doing so. Once this was completed, much of the other code was straightforward. Using the OctoPrint API just took some experimentation to find the right way to pass my program’s private API key into the message header, but once I figured that out, adding a file to its local directory and starting the print only takes about 12 lines of code (the snippet shown below assumes you saved the attachment into the same directory that you are running the script from):

One important thing to note is the line in printFile that reads “profile”: “cura”. OctoPrint has CuraEngine built in, so you need to download the Cura slicing software and configure some initial settings, then port your ‘.ini’ file into OctoPrint. OctoPrint only works with older versions of the Cura software, so you can download those, such as 15.04, here. On Mac, after configuration the path to your ‘.ini’ file is in “/your_username/Library/Application Support/cura/your_version/.” Just copy the file to your desktop and select it in OctoPrint’s settings –> Cura.

Another thing that took me a long time to do was making the program run in the background, waiting for new emails. Since I had originally written my program to scan an entire inbox (really not sure why), this meant I had to rewrite a lot of my methods to handle individual emails which really just resulted in me removing a lot of loops, consolidating methods, and generally writing cleaner code. One important thing that came out of this is that in order for the IMAP server to detect a new email as it comes in, it needs to reinitialize the inbox connection each time it searches. Furthermore, diligent closing and reopening of MySQLdb cursors between loops is necessary to prevent an error where the server thinks it is receiving multiple concurrent requests. Here is a snippet of code that checks for new emails, resetting connections afterwards and waiting a specified number of seconds between attempts:

All in all, I’m pretty happy with this so far, and am looking forward to creating a more streamlined product. To recap on progress made and forecast future happenings, below is an updated timeline of my DTSF experience.

Week 1:

  • Built 3D Printer, installed Repetier-Host

Week 2:

  • Initial test prints, loaded operating system onto Raspberry Pi (short week)

Week 3:

  • Tweaked printer settings, some bigger prints
  • Got basic temperature & humidity sensor running on Raspberry Pi, using basic graphing program (Flot)

Week 4:

  • Created an interactive website for my temperature and humidity sensor, learned graphing skills using Data Driven Documents (D3.js), and hosted the whole thing using Laravel 5 framework for PHP on Sites.
  • Learned SMTP functionality for Python, allowing my RPi to send emails when the temperature or humidity fell outside of a certain range.

Week 5:

  • Installed OctoPrint on my Pi, tried to build a plugin for the program but decided to use API calls and set a goals for my current project.

Week 6:

  • Got basic API calls working with OctoPrint, wrote Python program that could scan emails and download attachments, but could not take those attachments and upload them to OctoPrint.
  • Gained experience working with imaplib for Python

Week 7:

  • Program scans for emails, downloads them, checks for valid senders and attachments, grabs the attachments from valid emails, pushes them to OctoPrint, and starts the print job.

Week 8:

  • Start building a queue for pending print jobs and a website to monitor their progress.
  • Begin process of mirroring OctoPrint webcam stream on a Sites website, or simply an IP camera running on a HotSpot.

Week 9:

  • Add a physical button next to the printer that users can press to signal to the program that the current print has finished, and that it is safe to begin the next one.
  • Near completion on website, polishing and removing bugs.
  • Determine feasibility of Webcam stream, at very least lay down timeline for its development if it were unable to be completed within the 10-week timeframe, but plan on having it completed.

Week 10:

  • Complete a streamlined system where users and easily become authenticated, send emails to the printer, track and or watch their print progress, receive email updates upon print completion, and see the availability of printers in the West Building before choosing to print.

 

Week 5

Feels like we are finally approaching the main parts of our projects. Here’s the progress I made this week.

OctoPrint

Found at http://octoprint.org, OctoPrint is an open-source, remote-controlled 3-D printer software. With support for most modern printers, I was able to get it up and running very quickly. It has full support for both computer- and Raspberry Pi-based implementations, with instructions for the latter found at http://octopi.octoprint.org.

The software comes with built-in webcam support, progress reports, and a sizable repository of community developed plugins, meaning that a fair number of features that I had planned to develop for my printer already have fully fleshed out solutions. However, this is more encouraging than disparaging, and allows me to narrow my project focus as the program reaches its halfway point. Here is how I plan to move forward:

Developing a Gettysburg-Friendly Solution to Remote-Controlled Printing

The single biggest feature of OctoPrint, its ability to be controlled by any computer on the same network as its host server, is very tough to implement in environments like a college campus. Due to the sheer number of devices on a network, simply running an OctoPrint instance on localhost would be impractical, on top of being a large security vulnerability. One option would be to place the server on a public-facing website that requires the appropriate access credentials — however this would require more time than I would like, and I am interested in pursuing a different solution simply because I think it would be more fun. I would like a user to be able to simply send an email with an attachment containing what they want to print, and have the printer automatically print it.

Proposed Solution

OctoPrint comes with built-in API controls: found here, they make it possible for third-party sites or programs to access the OctoPrint instance without the need to physically navigate to the page and click buttons. Doing so requires an access key which is generated by the program, and every API request must contain this key in its header in order to be considered valid. Furthermore, these requests, formatted in the correct way, may be used to:

  • Load files into the printer queue
  • Slice them
  • Begin printing the file
  • Notify the user when the print is done

The best way I have found to do this so far is to use cURL: Assuming your OctoPrint instance is running on localhost:8080 and your API key is “yourkey”, a sample call to the OctoPrint API from the terminal, which would return a summary of connection settings, looks like this:

curl -H "X-Api-Key: yourkey" localhost:8080/api/connection

Where -H specifies the custom header you are providing which is required to be “X-Api-Key”.

So, what I propose is:

  • A python script, running concurrently on a Raspberry Pi hosting an OctoPrint instance, that holds login credentials to a single email account and uses IMAP to read its inbox (similar to the script from the temperature and humidity sensor, which used SMTP to send emails).
  • The script will contain a list of email addresses (possibly loaded from a database) which have been approved by an administrator to have access to the 3D printer.
  • Every couple of seconds or so, it will scan the inbox of the email account for emails from these pre-approved addresses.
  • Upon receiving an email from one of these addresses, it checks to see if it contains an attachment of filetype .stl.
    • Optional: For increased security, give each user a code which must be included in the body of the email in order for the program to run.
  • The script saves a copy of the file and calls a method which handles the printing process, passing the file as its parameter.
  • This method (likely with only a couple lines of code!) calls the OctoPrint API to
    • upload the file to the program,
    • select it for printing,
    • slice it into GCode, and
    • print the object.

This is admittedly fairly ambitious, but I think it is achievable and what I will be tackling next week.

Week 4

This week I spent almost entirely on making my Temperature and Humidity application better than it was before. Here is a screenshot of the finished product (also currently hosted here).

Now I’ll go into some of the specifics of what it took to get this together. Although I had already done some PHP for a different class, this ended up needing a bit more effort than I had thought it would.

The Framework

Due to firewall restrictions, I am unable to host a site directly from my own computer, so I was somewhat corralled into using Sites, which is what this blog is hosted on. Luckily, Sites is very comprehensive and comes with a ton of different options, so the only hindrance this created was that I had to learn some new stuff (which is what this program is about anyway). I selected Laravel 5 as my framework since I had heard it was fairly customizable. Laravel’s main draw is that it comes with a large library of functions and add-ons designed to make the web design process easier and more robust, but since I was making a small, nontraditional site, I didn’t need to use a lot of them. However, here is a brief overview of two of them that I didn’t know too much about before this week.

Routes

I have done routing before in Ruby, but Laravel declares them a different way. For the more familiar definition of routing, where a user travels to http://www.yourdomain.com/route and is directed to the designated page, it is done like so:

Where the view would be a PHP file named ‘yourview.php’ located in the resources/views/ directory. However, this only shows the user this page. If you wish to send data between pages on your site, you need to use a controller.

Controllers

In PHP, the most effective way to send data between a client to a server is by using GET and POST requests. In short, GET is used more often for nonsensitive information and usually returns variables from a function based on user input, while POST can be used to send encrypted information to the server to be processed. In my application, I needed to use a GET request to show a graph of data within a date range specified by the user. So, when the user clicks the button labelled “Get Selected Values” I needed to redirect that request to the controller which I wrote to query the database (I’ll get to the database in the next section) and return the correct values. So, in the code for the date range form, I have:

Meaning that when the button (which must be of ” type=’Submit’ “) is clicked, the information from the form is sent to a route called ‘fetch’. So, in my Routes.php, I have:

So that when a GET request is received at the ‘/fetch’ URI, Laravel accesses the controller (in the App/Http/Controllers/ directory) in the file called dateController, and uses the function named daterange. This is what the function looks like:

The important things to note here: Laravel handles database requests with the ‘DB::select’ function, and the credentials for the database are handled securely by the framework. The daterange function also requires an input of type Request (which is only available to use after importing the Illuminate\Http\Request resource), and the indexes of the request array are defined by the name of the html element from which that data came from. The datepicker elements don’t contain hours, minutes, or seconds, so we need to append a “00:00:00” string to the end so that they match the format of the database. Finally, we need to JSON encode the resulting PHP array so it will be readable when passed to our graphing function, which is written in Javascript.

Just to ensure all due credit is given, the datepicker I used in this application is available here. The CDN for its javascript/stylesheet requirements can also be found here. For some reason, it doesn’t work with all versions of JQuery. I was able to get it to work with the slim version of JQuery 3.1.1, but not with any version of JQuery 3.1.2.

The Database

Not a very complicated part of this project. Sites comes with PHPMyAdmin, which is pretty straightforward to use. I have one table, the primary key of which is the Date (to the second), and every entry also contains a Temperature and Humidity reading. The values are sent from my Raspberry Pi directly to the database, which is a fairly simple process done using the Python resource MySQLdb (link).

The Graph

The graph was created using a javascript library called Data Driven Documents, or D3.js (d3js.org). Also helpful was a free (pay-what-you-want) textbook which I found which is called D3 Tips and Tricks and is written by Malcolm McLean.

Important:

The newest version of D3 (v4) was released somewhat recently. This means that many code examples and tutorials you find online will be written for older versions, and the syntax for the two is NOT the same. The older versions are still available for use, but still proceed with caution. My app uses version 4.

D3 Tips and Tricks, V3.x (99% complete)

D3 Tips and Tricks, V4.x (66% complete as of 6/19/17)

While the syntax used by D3 can be fairly verbose at times, it does allow you total control over the data, and the code is easy enough to understand when you are reading it. Another great resource is http://bl.ocks.org, which contains tons of user-submitted code examples for D3. Mike Bostock, the creator of D3, also has some resources available on his personal blog at http://bost.ocks.org. The code for my graphing function is on my GitHub here, and was based off of code from the above textbooks, which is online here. The big difference between my graph and others is that my data does not come from a .csv (comma separated values) file, but instead from a JSON array derived from the database query output. This actually reduces the amount of code required, but was tough to figure out at first since there were not a ton of examples online. I have tried my best to clearly annotate my code.

The Email

The last feature I added to my program is the ability for my Python script (running on the Pi) to send an email when the temperature or humidity falls outside of a certain range. This was incredibly easy to do with SMTP using a Gmail account that I made specifically for this purpose. The Python resource I used is called SMTPLib (link), and my annotated script is on my GitHub here.

Next week, I plan on returning to work on the 3D printer. More updates to follow.

Week 2/3

After spending most of last week at home in Minnesota for my brother’s graduation, I will be posting two blog updates at once this week. We’ll start with my short Week 2.

Week 2

Tuesday, May 30 was the only day that I was on-campus. After all of the rebuilding efforts during Week 1, it was nice to see that it worked pretty much without incident the following week. I printed a few calibration cubes and spent a little while leveling the bed, then decided to see what I could get done with my Raspberry Pi. For the unaware, the Raspberry Pi is a small, cheap (~$30), open-source computer which can be used for a variety of applications and has the ability to interface with a large number of different sensors. If connected to a monitor and booted with a GUI-based operating system, it essentially can work the same way as a desktop computer. I have the newest version, which is called the Raspberry Pi 3 (hereafter referred to as Pi or RP3).

The Pi

The standard (and recommended) install for the RP3 is done using a 8GB (or larger) microSD card loaded with the officially supported installer, NOOBS (New Out Of Box Software), which sets up the Raspbian operating system on your Pi. Raspbian gets its name from Debian, a free, open source operating system on which it is based. This makes sense because Debian and other open source Unix-based operating systems tend to be popular with the same kind of people who also enjoy tinkering with things like the Pi. An added bonus is that macOS is also Unix-based, so many of Raspbian’s command line operations are the same as my usual OS. This proved very helpful, as I did not a sufficiently large microSD card available to hold the standard Raspbian install (current version at the time of writing: Raspbian Jessie). Instead, I used the Lite version of the operating system, which is a minimal version of the OS with almost nothing pre-installed. However, it fits nicely on a 2GB SD card!

Jessie Lite

To install the operating system on my Pi, I downloaded a .zip file of the official distribution onto my computer (found here), and used a free program called Etcher to flash (copy) it onto my SD card. After that, all you do is insert the SD card into the Pi, connect a keyboard, and display, and turn it on. However, after this I couldn’t do much. The RPi 3 comes with built in wifi capabilities, but even though I could connect to the campus wifi, I couldn’t use it to download any software or to connect to my Mac because the campus’ firewall was blocking all traffic coming to my device. I submitted the MAC address of my Pi to IT for (hopeful) approval, and did some cleanup before getting ready to head to the airport.

Week 3

After returning to campus, I got back to work on my Pi. I was still having issues with the wifi, but hooking up an ethernet cable worked perfectly. After running

    • ifconfig

To verify that my internet connection was working (if you see an IP address next to IPv4 under the section for ethernet connections, eth0, you are connected), and

    • sudo apt-get update

To update my software, I began following the instructions to create a temperature and humidity sensor described in the last chapter of Mahesh Venkitachalam’s Python Playground: Geeky Projects for the Curious Programmer. The goal of the project is to use the Pi to serve a webpage in which data from the sensor is plotted on a graph in real time. In a slight departure from Venkitachalam’s materials list, I used Adafruit’s 3-pin DHT-11 Temperature and Humidity Sensor (he used a different 4-pin version of the DHT-11). Venkitachalam also included a red LED bulb in his build, which could be controlled with a button on the webpage served by the Pi. I had no use for this so I didn’t do much with it after figuring out how to make it turn on and off on my own. Below is how to connect the sensor to the Pi using jump cables and a breadboard:

The textbook uses a very simple python module called Bottle to serve webpages. Here is the “Hello World!” example from their website:

The @route(‘/route’) method establishes a directory which the user may navigate to after the server has been started; the function immediately following the route declaration defines what the user actually sees when they visit the page. Strings inside angled brackets, like <name> in the example above, are placeholders for variables which can be used in various ways. The server is started with the function run(address, port) which defines the IP address and port at which the site will be served. In the above example, the user navigates to a subdirectory of “localhost:8080/hello/” corresponding to their own name: entering “localhost:8080/hello/James” will return a simple page saying “Hello James!” Other useful elements of the Bottle framework include ability to return static web pages (including stylesheets or javascript files for webpages) and the functionality to serve dynamic HTML.

Static webpages:

Serving HTML:

For graphing capabilities, Python Playground uses the Flot javascript library, documentation for which is found here.

I had to make several other adjustments to Venkitachalam’s code in order to make everything work properly. But it largely worked without issue:

Another important thing to note is that due to campus connectivity restrictions, I ended up having to share my macbook’s wifi connection with my Pi so that I could access localhost webpages it served. Instructions on how to do this can be found here. The full code from this project is posted on my GitHub.

Although I am happy with the above result, it is decidedly not very pretty and serves very little use as it is only accessible on my own machine. Furthermore, the bottle framework is not necessarily very secure. Moving forward I will find a better graphing solution and try to host the data collected by the Pi on my Sites, complete with email alerts if the temperature or humidity were to fall outside a given range.

Now on to some 3D printing. I was able to get some (relatively) larger prints done, such as this boat.

As you can see, the quality is obviously not super great. Here’s why.

My extrusion width is too high.

This boat was created with a first layer extrusion width of .6mm and subsequent layer width of .5mm. This allows for faster prints, as less layers are needed to fill the same space, but at the expense of quality. After doing tests at widths of .2 and .1mm, I immediately saw an increase in build quality. However, my prints were still getting large globs around the edges, and way too much filament was oozing out of the extruder head–it was evening happening during moves that didn’t require any printing.

My temperature is too high.

My filament–Hatchbox White PLA– has a recommended temperature range of 180-210 degrees Celsius. At first, I had been printing at 215 degrees Celsius. Some online searches of others’ experiences with the same filament found indications that temperatures on the lower end of the recommended range worked best, and I eventually received my best results at a temperature of 175 degrees Celsius. I suspect that this is not necessarily accurate, as the fan designed to regulate the temperature of the extruder head is also blowing directly on the temperature sensor. I have purchased some other temperature sensors and plan on testing this hypothesis soon.

My extruder is extruding too fast, retracting too much filament, and moving too slowly.

I played with a laundry list of printer settings before this combination of variables came to light as the likely culprit for the globs accumulating at the corners of my prints. I ran a 4 Cube Retraction Calibration test (link), which prints cubes of varying distances away from each other on the same plane. Since the extruder has to move travel fairly long distances between layers in order to keep the cubes level with each other, it emphasizes over- or under- retraction as the former causes the extruder to stay in one place for too long, oozing too much excess filament in the process, and the latter simply does not effectively clear filament from the extruder head. It is important to note that once filament has reached the last part of the extruder, it cannot actually be fully retracted again while the printer is on since it has already melted. Thus, the speed of non-printing moves needs to be high (eventually I settled on around 300 mm/s), and extrusion/retraction settings need to be finely tuned. Below are selected pictures of my extrusion settings and the resulting cube tests (some intermediary tests not shown as I tried to only change one variable at a time).

Getting better, but not quite perfect yet. A more detailed description of my printer settings will be posted here once I get something I am really happy with. I have been keeping records of my tests in a notebook–these will be documented in a separate blog I am working on and will eventually be posted here as well.

Next Week

On Friday, Eric helped me fix some access issues I was having with my own personal Sites page. After this was resolved, I now have slightly more control over what I can place on my own Sites. This will help in creating more secure applications, and I am currently working on a much more in-depth version of my temperature and humidity sensor application. The goals for this application, which I hope to have done by next week:

  • A live, accessible webpage which reads data from the Raspberry Pi and updates it on a prettier graph.
    • I have decided to try to use D3js for my graphical purposes. This blog by its creator details implementation of real-time graphs using random data, so I am going to try to alter this to plot data received from my Pi.
      • As of Friday, already have a Sites page built on Laravel running a random-data version.
  • Email alerts using a php script that will send alerts if the temperature measured by the sensor goes out of a certain range.
  • Implementation of a database for trend analysis.

Aside from this, my next two tasks involve the 3d printer and are to use software called OctoPrint to wirelessly interface with my printer, and to begin improving its sensory inputs with additional temperature sensors and readouts.

 

DTSF Blog Post Week 1 – James

Introduction

I’m James Arps, Chanhassen, MN native and Gettysburg College Math & Computer Science double major. I graduate in the spring of 2018, and in addition to my studies and the DTSF fellowship this summer, I am also involved in Gettysburg’s Interfraternity Council as the Vice President of Management and am the Risk Manager of Phi Delta Theta, Pennsylvania Beta. I plan on applying to graduate programs in Computer Science this fall, and this summer not only presents an opportunity to hone my skills in an area of high interest to me, but also allows me to give back to a college which has provided me with such a large potential for growth and personal development.

Goal setting

By the end of this program, I hope to develop a streamlined method through which faculty or their students may use a web browser to remotely access 3-D printers located in the campus’ West Building, upload build schematics, watch the printing progress through an IPCamera utilizing an Intel Edison, and receive alerts via text, tweet, or email once their item has finished printing, or (should extenuating circumstances occur) be notified if their build fails to print. Included in this plan are a few intermediate goals, as well as plans for the development of a scheduling system which would track and show which printers are currently in use, while potentially allowing the user to autonomously begin their printing job during busy hours if a printer is currently not available.

Week 1:

On Monday, May 22, the other two fellows and I were introduced to some basic guidelines which would be governing our summer experience, and a basic 3-D printer kit was given to myself and Jonathan. This kit, named the FLSun-7, will form the basis of my project, and one of my objectives is to improve the quality of its outputs. At a base cost of $250, even slight improvements made using relatively cheap technology will put its usefulness at the level of printers two or three times its price. I plan on doing this by using an external chip to add additional fans and temperature sensors, as well adding as improved monitoring capabilities. However, as I quickly found out, all assembly is required with these kits, and progress did not necessarily come as quickly as expected.

By Wednesday, I was a little more than halfway through the approximately 150-page instruction manual. I had a basic frame, with a couple motors and few sensors attached. The gray panel shown is the heated printing bed, which we later covered in tape and glue to provide a more stable and adhesive printing surface.

I also installed the program Repetier-host on my computer, which allows my Macbook to interface with the printer.  I am a Mac user, and prefer to use the terminal to install new programs. I installed Repetier-host through Homebrew, which can be done as follows:

  • Install Homebrew (OSX’s “missing package manager”) by pasting the following into your terminal:
    • /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • Install brew-cask by pasting the following into your terminal:
    • brew install homebrew-cask-completion
      
  • Install Repetier-host by pasting the following into your terminal:
    • brew cask install repetier-host-mac

 

On Thursday, I ran into a serious problem. Although I had thought I had been checking the orientation of every part I installed against the manual, I had accidentally oriented the extruder the wrong way:

As seen here, the extruder would be extruding filament completely off of the build plate. The fix for this required an entire reassembly of the machine’s x-axis and a few hours. The final, finished product is shown below:

We also attended multiple education sessions this week on various subjects such as documentation, project management, and website development. These were done in conjunction with Gettysburg College’s Digital Summer Scholarship Fellows, and we will continue to work with them in the future. A brief outline of my future plans are as follows.

Outline

Week 2:

  • Use a Raspberry Pi to develop a temperature and humidity sensor for the library’s special collections sensor.

Week 3:

  • Finish online interface for this sensor, perfect 3-D printing techniques using base FL-Sun7 3-D model.

Week 4-5:

  • Add additional temperature sensors and fans to FL-Sun7 3-D, either using raspberry pi or Intel Edison. Do any other work necessary to improve printing output.

Week 6:

Week 7:

  • Use node.js, existing Gettysburg technology to move the site created during the previous week into the public domain and integrate database support to allow traditional college logins to access the site.
  • Begin implementing scheduling system, CSS updates to make the website usable and pretty.

Week 8:

  • Bug testing, functionality, UI improvements.

Week 9-10:

  • Add additional features as time allows and heavily focus on providing a streamlined, error-free experience which can be easily presented on and learned from through extensive documentation.