Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3707 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.2370. Min Act: -0.0000

Data Index: 839428 (The Pile)

Max Activating Token Index: 611

Text #1

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1865. Min Act: -0.0000

Data Index: 764856 (The Pile)

Max Activating Token Index: 768

Text #2

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1857. Min Act: -0.0000

Data Index: 855918 (The Pile)

Max Activating Token Index: 413

Text #3

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1853. Min Act: -0.0000

Data Index: 738382 (The Pile)

Max Activating Token Index: 721

Text #4

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1949. Min Act: -0.0000

Data Index: 868247 (The Pile)

Max Activating Token Index: 906

Text #5

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1818. Min Act: -0.0000

Data Index: 654208 (The Pile)

Max Activating Token Index: 79

Text #6

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1771. Min Act: -0.0000

Data Index: 457034 (The Pile)

Max Activating Token Index: 757

Text #7

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1587. Min Act: -0.0000

Data Index: 1873092 (The Pile)

Max Activating Token Index: 20

Text #8

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1641. Min Act: -0.0000

Data Index: 860826 (The Pile)

Max Activating Token Index: 324

Text #9

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1671. Min Act: -0.0000

Data Index: 1050137 (The Pile)

Max Activating Token Index: 369

Text #10

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1618. Min Act: -0.0000

Data Index: 462045 (The Pile)

Max Activating Token Index: 547

Text #11

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1521. Min Act: -0.0000

Data Index: 289797 (The Pile)

Max Activating Token Index: 835

Text #12

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1665. Min Act: -0.0000

Data Index: 509617 (The Pile)

Max Activating Token Index: 504

Text #13

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1569. Min Act: -0.0000

Data Index: 879962 (The Pile)

Max Activating Token Index: 425

Text #14

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1410. Min Act: -0.0000

Data Index: 1105174 (The Pile)

Max Activating Token Index: 470

Text #15

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1527. Min Act: -0.0000

Data Index: 1770771 (The Pile)

Max Activating Token Index: 532

Text #16

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1545. Min Act: -0.0000

Data Index: 1384033 (The Pile)

Max Activating Token Index: 273

Text #17

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1446. Min Act: -0.0000

Data Index: 1550361 (The Pile)

Max Activating Token Index: 834

Text #18

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1256. Min Act: -0.0000

Data Index: 1938390 (The Pile)

Max Activating Token Index: 601

Text #19

Max Range: 0.2370. Min Range: -0.2370

Max Act: 0.1517. Min Act: -0.0000

Data Index: 1330220 (The Pile)

Max Activating Token Index: 149