Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3427 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.5384. Min Act: -0.0001

Data Index: 1570730 (Python Code)

Max Activating Token Index: 687

Text #1

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.5440. Min Act: -0.0001

Data Index: 1402468 (Python Code)

Max Activating Token Index: 372

Text #2

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4390. Min Act: -0.0001

Data Index: 1618683 (Python Code)

Max Activating Token Index: 342

Text #3

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.5689. Min Act: -0.0001

Data Index: 1481220 (Python Code)

Max Activating Token Index: 966

Text #4

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.5192. Min Act: -0.0001

Data Index: 1563938 (Python Code)

Max Activating Token Index: 704

Text #5

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4617. Min Act: -0.0001

Data Index: 1561686 (Python Code)

Max Activating Token Index: 26

Text #6

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4903. Min Act: -0.0001

Data Index: 1580575 (Python Code)

Max Activating Token Index: 548

Text #7

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4747. Min Act: -0.0001

Data Index: 1588873 (Python Code)

Max Activating Token Index: 280

Text #8

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4335. Min Act: -0.0001

Data Index: 1637910 (Python Code)

Max Activating Token Index: 995

Text #9

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4284. Min Act: -0.0001

Data Index: 1507231 (Python Code)

Max Activating Token Index: 535

Text #10

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4260. Min Act: -0.0001

Data Index: 1618876 (Python Code)

Max Activating Token Index: 1012

Text #11

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4748. Min Act: -0.0001

Data Index: 1631834 (Python Code)

Max Activating Token Index: 564

Text #12

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4984. Min Act: -0.0001

Data Index: 1553925 (Python Code)

Max Activating Token Index: 783

Text #13

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4478. Min Act: -0.0001

Data Index: 1518923 (Python Code)

Max Activating Token Index: 560

Text #14

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4821. Min Act: -0.0001

Data Index: 1487845 (Python Code)

Max Activating Token Index: 1015

Text #15

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4252. Min Act: -0.0001

Data Index: 1458341 (Python Code)

Max Activating Token Index: 291

Text #16

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4286. Min Act: -0.0001

Data Index: 1377200 (Python Code)

Max Activating Token Index: 108

Text #17

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4189. Min Act: -0.0001

Data Index: 1577021 (Python Code)

Max Activating Token Index: 542

Text #18

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4574. Min Act: -0.0001

Data Index: 1593062 (Python Code)

Max Activating Token Index: 683

Text #19

Max Range: 1.5689. Min Range: -1.5689

Max Act: 1.4051. Min Act: -0.0001

Data Index: 1416291 (Python Code)

Max Activating Token Index: 768