Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2689 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0063. Min Range: -3.0063

Max Act: 3.0063. Min Act: -0.0000

Data Index: 1557641 (Python Code)

Max Activating Token Index: 446

Text #1

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.9595. Min Act: 0.0000

Data Index: 1629916 (Python Code)

Max Activating Token Index: 115

Text #2

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.7583. Min Act: -0.0000

Data Index: 1400921 (Python Code)

Max Activating Token Index: 97

Text #3

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.5734. Min Act: -0.0000

Data Index: 1439977 (Python Code)

Max Activating Token Index: 626

Text #4

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.5358. Min Act: -0.0000

Data Index: 1415391 (Python Code)

Max Activating Token Index: 810

Text #5

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.6601. Min Act: -0.0000

Data Index: 1608893 (Python Code)

Max Activating Token Index: 664

Text #6

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.3911. Min Act: 0.0000

Data Index: 1472778 (Python Code)

Max Activating Token Index: 199

Text #7

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.3751. Min Act: -0.0000

Data Index: 1477815 (Python Code)

Max Activating Token Index: 570

Text #8

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.3125. Min Act: -0.0000

Data Index: 1508638 (Python Code)

Max Activating Token Index: 519

Text #9

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.2349. Min Act: -0.0000

Data Index: 1493716 (Python Code)

Max Activating Token Index: 769

Text #10

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.1433. Min Act: 0.0001

Data Index: 1525163 (Python Code)

Max Activating Token Index: 307

Text #11

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.2210. Min Act: 0.0001

Data Index: 1588687 (Python Code)

Max Activating Token Index: 422

Text #12

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.1880. Min Act: 0.0000

Data Index: 1582002 (Python Code)

Max Activating Token Index: 591

Text #13

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.2873. Min Act: -0.0000

Data Index: 1395057 (Python Code)

Max Activating Token Index: 739

Text #14

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.1386. Min Act: -0.0000

Data Index: 1370884 (Python Code)

Max Activating Token Index: 737

Text #15

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.0224. Min Act: -0.0000

Data Index: 1444286 (Python Code)

Max Activating Token Index: 820

Text #16

Max Range: 3.0063. Min Range: -3.0063

Max Act: 1.9781. Min Act: -0.0000

Data Index: 1634355 (Python Code)

Max Activating Token Index: 606

Text #17

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.2111. Min Act: -0.0000

Data Index: 1420309 (Python Code)

Max Activating Token Index: 511

Text #18

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.0448. Min Act: -0.0000

Data Index: 1574183 (Python Code)

Max Activating Token Index: 137

Text #19

Max Range: 3.0063. Min Range: -3.0063

Max Act: 2.0680. Min Act: -0.0000

Data Index: 1526059 (Python Code)

Max Activating Token Index: 1013