Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4085 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1951. Min Act: -0.0002

Data Index: 99577 (C4 (Web Text))

Max Activating Token Index: 134

Text #1

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1787. Min Act: -0.0002

Data Index: 198309 (C4 (Web Text))

Max Activating Token Index: 143

Text #2

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1826. Min Act: -0.0002

Data Index: 1138842 (C4 (Web Text))

Max Activating Token Index: 130

Text #3

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1675. Min Act: -0.0002

Data Index: 509115 (C4 (Web Text))

Max Activating Token Index: 651

Text #4

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1644. Min Act: -0.0002

Data Index: 1079103 (C4 (Web Text))

Max Activating Token Index: 479

Text #5

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1590. Min Act: -0.0002

Data Index: 869073 (C4 (Web Text))

Max Activating Token Index: 789

Text #6

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1488. Min Act: -0.0002

Data Index: 1304459 (C4 (Web Text))

Max Activating Token Index: 615

Text #7

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1445. Min Act: -0.0002

Data Index: 751680 (C4 (Web Text))

Max Activating Token Index: 734

Text #8

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1441. Min Act: -0.0002

Data Index: 1136074 (C4 (Web Text))

Max Activating Token Index: 648

Text #9

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1436. Min Act: -0.0002

Data Index: 9990 (C4 (Web Text))

Max Activating Token Index: 767

Text #10

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1422. Min Act: -0.0002

Data Index: 540568 (C4 (Web Text))

Max Activating Token Index: 115

Text #11

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1443. Min Act: -0.0002

Data Index: 371912 (C4 (Web Text))

Max Activating Token Index: 264

Text #12

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1398. Min Act: -0.0002

Data Index: 1078973 (C4 (Web Text))

Max Activating Token Index: 442

Text #13

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1307. Min Act: -0.0002

Data Index: 990702 (C4 (Web Text))

Max Activating Token Index: 701

Text #14

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1331. Min Act: -0.0002

Data Index: 664269 (C4 (Web Text))

Max Activating Token Index: 830

Text #15

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1347. Min Act: -0.0002

Data Index: 354697 (C4 (Web Text))

Max Activating Token Index: 775

Text #16

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1343. Min Act: -0.0002

Data Index: 550675 (C4 (Web Text))

Max Activating Token Index: 789

Text #17

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1324. Min Act: -0.0002

Data Index: 422622 (C4 (Web Text))

Max Activating Token Index: 777

Text #18

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1282. Min Act: -0.0002

Data Index: 421034 (C4 (Web Text))

Max Activating Token Index: 668

Text #19

Max Range: 0.1951. Min Range: -0.1951

Max Act: 0.1265. Min Act: -0.0002

Data Index: 104975 (C4 (Web Text))

Max Activating Token Index: 839