OpenGL selection a region of a .stl file

I am looking to develop an OpenGL application for a desktop environment that can take a .stl file and allow the user to select a region. The key functionality would be the ability to select a region of the .stl file and have it highlighted with a color. Please note that the .stl files may have triangles behind the selected surface area. those triangles should not be selected. The attached video shows the operation and expectation is to speed up the selection of the triangle. Current algorithm: Our current algorithm creates a rectangular region. It selects all triangles which are inside the region ( all triangles in the near plane, intermediate planes and far plane are selected). we have an O(N^2) algorithm which checks the distance of each of triangle in relationship with another and identify the closest triangles and colors them. Improvement: This algorithm is very slow for dense triangles. In OpenGL, we are sure there are functions which will immediately select a region of a triangle. Expectation: Source code and explanation of how to improve selection of a region of an .stl file. Provide an O(N) solution to select a region. E.g. If OpenGL has a function which helps capture the visible triangles in the graph, this could be done.