Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1747 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.1273. Min Range: -4.1273

Max Act: 4.1273. Min Act: -0.0001

Data Index: 1492494 (Python Code)

Max Activating Token Index: 37

Text #1

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.9976. Min Act: -0.0000

Data Index: 1554629 (Python Code)

Max Activating Token Index: 326

Text #2

Max Range: 4.1273. Min Range: -4.1273

Max Act: 3.0376. Min Act: -0.0001

Data Index: 1648162 (Python Code)

Max Activating Token Index: 43

Text #3

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.6984. Min Act: -0.0000

Data Index: 1542916 (Python Code)

Max Activating Token Index: 775

Text #4

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.7313. Min Act: -0.0001

Data Index: 1649204 (Python Code)

Max Activating Token Index: 111

Text #5

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.5304. Min Act: -0.0001

Data Index: 1593470 (Python Code)

Max Activating Token Index: 845

Text #6

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.8093. Min Act: -0.0001

Data Index: 1578302 (Python Code)

Max Activating Token Index: 818

Text #7

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.4713. Min Act: -0.0000

Data Index: 1470711 (Python Code)

Max Activating Token Index: 357

Text #8

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.5470. Min Act: -0.0001

Data Index: 1631918 (Python Code)

Max Activating Token Index: 138

Text #9

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.5287. Min Act: -0.0001

Data Index: 1437384 (Python Code)

Max Activating Token Index: 852

Text #10

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.3939. Min Act: -0.0000

Data Index: 1539262 (Python Code)

Max Activating Token Index: 570

Text #11

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.4632. Min Act: -0.0001

Data Index: 1385386 (Python Code)

Max Activating Token Index: 681

Text #12

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.5240. Min Act: -0.0000

Data Index: 1521159 (Python Code)

Max Activating Token Index: 221

Text #13

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.2464. Min Act: -0.0000

Data Index: 1637625 (Python Code)

Max Activating Token Index: 34

Text #14

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.3715. Min Act: -0.0001

Data Index: 1535585 (Python Code)

Max Activating Token Index: 406

Text #15

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.2136. Min Act: -0.0001

Data Index: 1394606 (Python Code)

Max Activating Token Index: 206

Text #16

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.2362. Min Act: -0.0000

Data Index: 1543471 (Python Code)

Max Activating Token Index: 9

Text #17

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.2057. Min Act: -0.0000

Data Index: 1595102 (Python Code)

Max Activating Token Index: 407

Text #18

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.2001. Min Act: -0.0001

Data Index: 1383072 (Python Code)

Max Activating Token Index: 258

Text #19

Max Range: 4.1273. Min Range: -4.1273

Max Act: 2.1802. Min Act: -0.0001

Data Index: 1651118 (Python Code)

Max Activating Token Index: 612