Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2354 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2583. Min Act: -0.0000

Data Index: 881578 (C4 (Web Text))

Max Activating Token Index: 706

Text #1

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2671. Min Act: -0.0000

Data Index: 1434527 (Python Code)

Max Activating Token Index: 961

Text #2

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2367. Min Act: -0.0000

Data Index: 1454223 (Python Code)

Max Activating Token Index: 20

Text #3

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2383. Min Act: -0.0000

Data Index: 1477536 (Python Code)

Max Activating Token Index: 332

Text #4

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2410. Min Act: -0.0000

Data Index: 1510759 (Python Code)

Max Activating Token Index: 989

Text #5

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2393. Min Act: -0.0000

Data Index: 1401563 (Python Code)

Max Activating Token Index: 621

Text #6

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2282. Min Act: -0.0000

Data Index: 1457594 (Python Code)

Max Activating Token Index: 37

Text #7

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2160. Min Act: -0.0000

Data Index: 1647223 (Python Code)

Max Activating Token Index: 324

Text #8

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2294. Min Act: -0.0000

Data Index: 1125622 (C4 (Web Text))

Max Activating Token Index: 863

Text #9

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2034. Min Act: -0.0000

Data Index: 1444671 (Python Code)

Max Activating Token Index: 871

Text #10

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2131. Min Act: -0.0000

Data Index: 372284 (C4 (Web Text))

Max Activating Token Index: 381

Text #11

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2194. Min Act: -0.0000

Data Index: 1055968 (C4 (Web Text))

Max Activating Token Index: 19

Text #12

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.1970. Min Act: -0.0000

Data Index: 330904 (C4 (Web Text))

Max Activating Token Index: 282

Text #13

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2168. Min Act: -0.0000

Data Index: 1233692 (C4 (Web Text))

Max Activating Token Index: 545

Text #14

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2087. Min Act: -0.0000

Data Index: 1571774 (Python Code)

Max Activating Token Index: 31

Text #15

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2132. Min Act: -0.0000

Data Index: 1071608 (C4 (Web Text))

Max Activating Token Index: 546

Text #16

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2001. Min Act: -0.0000

Data Index: 767210 (C4 (Web Text))

Max Activating Token Index: 891

Text #17

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.1929. Min Act: -0.0000

Data Index: 1432257 (Python Code)

Max Activating Token Index: 738

Text #18

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2029. Min Act: -0.0000

Data Index: 281965 (C4 (Web Text))

Max Activating Token Index: 17

Text #19

Max Range: 0.2671. Min Range: -0.2671

Max Act: 0.2031. Min Act: -0.0000

Data Index: 1601105 (Python Code)

Max Activating Token Index: 597