Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2642 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 6.8434. Min Range: -6.8434

Max Act: 6.8434. Min Act: -0.0000

Data Index: 1429291 (Python Code)

Max Activating Token Index: 641

Text #1

Max Range: 6.8434. Min Range: -6.8434

Max Act: 6.0270. Min Act: -0.0000

Data Index: 1515376 (Python Code)

Max Activating Token Index: 595

Text #2

Max Range: 6.8434. Min Range: -6.8434

Max Act: 6.0831. Min Act: -0.0000

Data Index: 1628726 (Python Code)

Max Activating Token Index: 1010

Text #3

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.7166. Min Act: -0.0000

Data Index: 1471037 (Python Code)

Max Activating Token Index: 766

Text #4

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.7469. Min Act: -0.0000

Data Index: 1436547 (Python Code)

Max Activating Token Index: 135

Text #5

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.3933. Min Act: -0.0000

Data Index: 1417145 (Python Code)

Max Activating Token Index: 533

Text #6

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.3348. Min Act: -0.0000

Data Index: 1563955 (Python Code)

Max Activating Token Index: 881

Text #7

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.5261. Min Act: -0.0000

Data Index: 1649528 (Python Code)

Max Activating Token Index: 618

Text #8

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.5267. Min Act: -0.0000

Data Index: 1415403 (Python Code)

Max Activating Token Index: 989

Text #9

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.3178. Min Act: -0.0000

Data Index: 1595277 (Python Code)

Max Activating Token Index: 630

Text #10

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.4001. Min Act: -0.0000

Data Index: 1498758 (Python Code)

Max Activating Token Index: 349

Text #11

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.0230. Min Act: -0.0000

Data Index: 1403478 (Python Code)

Max Activating Token Index: 836

Text #12

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.1789. Min Act: -0.0000

Data Index: 1518147 (Python Code)

Max Activating Token Index: 437

Text #13

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.5341. Min Act: -0.0000

Data Index: 1427115 (Python Code)

Max Activating Token Index: 781

Text #14

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.1894. Min Act: -0.0000

Data Index: 1512984 (Python Code)

Max Activating Token Index: 796

Text #15

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.3452. Min Act: -0.0000

Data Index: 1404921 (Python Code)

Max Activating Token Index: 409

Text #16

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.1313. Min Act: -0.0000

Data Index: 1631765 (Python Code)

Max Activating Token Index: 243

Text #17

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.2378. Min Act: -0.0000

Data Index: 1491690 (Python Code)

Max Activating Token Index: 651

Text #18

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.3881. Min Act: -0.0000

Data Index: 1556497 (Python Code)

Max Activating Token Index: 254

Text #19

Max Range: 6.8434. Min Range: -6.8434

Max Act: 5.3610. Min Act: -0.0000

Data Index: 1618822 (Python Code)

Max Activating Token Index: 973