Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4162 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.8708. Min Act: -0.0000

Data Index: 1418199 (Python Code)

Max Activating Token Index: 500

Text #1

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.8606. Min Act: -0.0000

Data Index: 1408012 (Python Code)

Max Activating Token Index: 1016

Text #2

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.8228. Min Act: -0.0000

Data Index: 1463318 (Python Code)

Max Activating Token Index: 670

Text #3

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.8274. Min Act: -0.0000

Data Index: 1506393 (Python Code)

Max Activating Token Index: 925

Text #4

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.7495. Min Act: -0.0000

Data Index: 1620014 (Python Code)

Max Activating Token Index: 777

Text #5

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.7343. Min Act: -0.0000

Data Index: 1641594 (Python Code)

Max Activating Token Index: 716

Text #6

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.7595. Min Act: -0.0000

Data Index: 1568805 (Python Code)

Max Activating Token Index: 646

Text #7

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.7391. Min Act: -0.0000

Data Index: 1632974 (Python Code)

Max Activating Token Index: 852

Text #8

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6764. Min Act: -0.0000

Data Index: 1422010 (Python Code)

Max Activating Token Index: 499

Text #9

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.7206. Min Act: -0.0000

Data Index: 1522662 (Python Code)

Max Activating Token Index: 929

Text #10

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6784. Min Act: -0.0000

Data Index: 1471908 (Python Code)

Max Activating Token Index: 678

Text #11

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.7139. Min Act: -0.0000

Data Index: 1407746 (Python Code)

Max Activating Token Index: 575

Text #12

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.7027. Min Act: -0.0000

Data Index: 1463892 (Python Code)

Max Activating Token Index: 734

Text #13

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6479. Min Act: -0.0000

Data Index: 1556643 (Python Code)

Max Activating Token Index: 736

Text #14

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6520. Min Act: -0.0000

Data Index: 1624739 (Python Code)

Max Activating Token Index: 654

Text #15

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6466. Min Act: -0.0000

Data Index: 1370058 (Python Code)

Max Activating Token Index: 350

Text #16

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6604. Min Act: -0.0000

Data Index: 1393399 (Python Code)

Max Activating Token Index: 644

Text #17

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.5909. Min Act: -0.0000

Data Index: 1588130 (Python Code)

Max Activating Token Index: 513

Text #18

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6082. Min Act: -0.0000

Data Index: 1465369 (Python Code)

Max Activating Token Index: 602

Text #19

Max Range: 0.8708. Min Range: -0.8708

Max Act: 0.6356. Min Act: -0.0000

Data Index: 1468926 (Python Code)

Max Activating Token Index: 366