Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 824 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 2.0458. Min Range: -2.0458

Max Act: 2.0458. Min Act: -0.0000

Data Index: 1649649 (Python Code)

Max Activating Token Index: 747

Text #1

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.9052. Min Act: -0.0000

Data Index: 1504141 (Python Code)

Max Activating Token Index: 273

Text #2

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.8987. Min Act: -0.0000

Data Index: 1403134 (Python Code)

Max Activating Token Index: 306

Text #3

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.9520. Min Act: -0.0000

Data Index: 1598438 (Python Code)

Max Activating Token Index: 133

Text #4

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.7790. Min Act: -0.0000

Data Index: 1504804 (Python Code)

Max Activating Token Index: 492

Text #5

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.8090. Min Act: -0.0000

Data Index: 1434735 (Python Code)

Max Activating Token Index: 317

Text #6

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.8350. Min Act: -0.0000

Data Index: 1554053 (Python Code)

Max Activating Token Index: 629

Text #7

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.7747. Min Act: -0.0000

Data Index: 1424272 (Python Code)

Max Activating Token Index: 329

Text #8

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6999. Min Act: -0.0000

Data Index: 1386316 (Python Code)

Max Activating Token Index: 169

Text #9

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6414. Min Act: -0.0000

Data Index: 1653039 (Python Code)

Max Activating Token Index: 126

Text #10

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.7037. Min Act: -0.0000

Data Index: 1543530 (Python Code)

Max Activating Token Index: 35

Text #11

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.7053. Min Act: -0.0000

Data Index: 1483545 (Python Code)

Max Activating Token Index: 132

Text #12

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6488. Min Act: -0.0000

Data Index: 1444231 (Python Code)

Max Activating Token Index: 51

Text #13

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.7498. Min Act: -0.0000

Data Index: 1518860 (Python Code)

Max Activating Token Index: 683

Text #14

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6533. Min Act: -0.0000

Data Index: 1489025 (Python Code)

Max Activating Token Index: 681

Text #15

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6309. Min Act: -0.0000

Data Index: 1387186 (Python Code)

Max Activating Token Index: 720

Text #16

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6779. Min Act: -0.0000

Data Index: 1524130 (Python Code)

Max Activating Token Index: 272

Text #17

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6195. Min Act: -0.0000

Data Index: 1576832 (Python Code)

Max Activating Token Index: 87

Text #18

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6542. Min Act: -0.0000

Data Index: 1479431 (Python Code)

Max Activating Token Index: 178

Text #19

Max Range: 2.0458. Min Range: -2.0458

Max Act: 1.6417. Min Act: -0.0000

Data Index: 1391420 (Python Code)

Max Activating Token Index: 160